Oracle Database 11g Release High Availability About the Authors Scott Jesse (Colorado Springs, Colorado) has worked for Oracle since 1996 and is currently Customer Support Senior Manager for the Oracle RAC Assurance Team He has worked with the Oracle RAC Assurance Team since 2007 Prior to this assignment, Scott’s primary focus was on clustering technologies He has served as a Global Team Lead and as a member of the Advanced Resolution and Escalations Team within Oracle Support, providing him with a unique insight into the needs of Oracle technology customers Scott is the coauthor of Oracle Database 10g High Availability with RAC, Flashback, and Data Guard and Oracle9i for Windows 2000 Tips & Techniques, both by McGraw-Hill/Oracle Press Bill Burton (United Kingdom) joined Oracle from Computer Associates in 1998, spending two years with the Oracle UK product support group for Database before transferring to the U.S Gold Support Team in October 2000 He spent time in the Oracle HA Support group, dealing with Oracle RAC and Oracle Data Guard, before joining the Bug Determination and Escalation group (BDE) for those products He has been a member of the Oracle RAC Assurance Development Team since 2007 Together with Josh Ort, he wrote and presented the Upgrading to 11gR2 Session at Oracle OpenWorld 2010 Bryan Vongray (Beaver, Pennsylvania) has more than nine years of experience implementing and supporting all aspects of the Oracle Database He specialized in Oracle High Availability with specific focus in Oracle Real Application Clusters (Oracle RAC), Oracle Data Guard, Streams, backup and recovery, as well as Grid Control As a Senior Consultant for Oracle Consulting Services, Bryan had planned and implemented Oracle Maximum Availability Architecture (MAA) Solutions for numerous Oracle customers Bryan is now putting his Oracle MAA expertise to use as a member of the Oracle RAC Assurance Team within Oracle Global Customer Support About the Contributor Hagen Herbst (Germany) is an OCP-certified DBA with more than a decade of experience in Oracle products Prior to joining the Oracle RAC Assurance Team within Oracle Global Customer Support, he spent years installing and upgrading countless databases, mainly Oracle RAC and Failsafe Systems in Europe, the Middle East, and Africa Hagen has also served as the senior Oracle DBA in a data center, utilizing all aspects of Oracle’s Maximum Available Architecture About the Technical Editors Farouk Abushaban is a Senior Principal Engineer at Oracle He is a founding member of the Center of Excellence Team and is the Global Technical Lead for Enterprise Manager High Availability implementations He spent 22 years in customer support and information systems Farouk works directly with strategic Oracle customers and partners to improve product utilization and performance in large deployments He has a bachelor’s degree in Electrical Engineering from the University of Colorado in Colorado Springs and completed his master’s degree in Management of Information Technologies with emphasis on Project Management and Security from Colorado Technical University Farouk has been with Oracle since 1997 Jiong Dai (Daedal) has worked with Oracle for more than ten years He started in Advanced Product Support, where he supported Oracle RDBMS, Oracle Developer, Oracle Designer, and Oracle Application Server, delivering onsite OPS design, Oracle backup/recovery strategy review, and database performance review He became an Oracle Certified Professional in 2000 In 2004, he worked with Intel Server Performance Scalability Lab, publishing the world’s leading TPC-H results and world leading MAPI Messaging Benchmark (MMB3) results, and working for the enterprise workload character analysis on TPC-C, SPECjAppServer (Java Application Server), SJBB, and so on In 2006, he moved to the Oracle HA Product Support Team and eventually moved on to the Oracle RAC Assurance Team in 2009 Currently, he provides support to critical Oracle RAC customers to assure their successful implementation on Oracle RAC He is the author of RAC Assurance Support Team: RAC Starter Kit and Best Practices (AIX) (Linux) Michael Smith is a principal member of the technical staff in Oracle’s Maximum Availability Architecture (MAA) Team in Server Technologies Mike has been with Oracle for 12 years, previously serving as the Oracle Data Guard Global Technical Lead within Oracle Global Support Mike’s current focus is developing, validating, and publishing HA best practices using Oracle Data Guard in an integrated fashion across all Oracle Database HA features His Oracle Data Guard technical specialties focus on network transport, recovery, role transitions, Oracle Active Data Guard, and client failover He has published a dozen MAA Best Practice Papers for Oracle 9i, 10g, and 11g He has been a contributing author to previous Oracle Press publications Mike has also been speaker at three Oracle OpenWorld events held in San Francisco His “What They Didn’t Print in the DOC” Best Practice Presentations covering Oracle Data Guard and MAA are a favorite among Oracle users, with attendance at the top of all Oracle Database technology presentations Paul Tjhang, Principal Technical Support Engineer, Oracle Corporation, worked as a system administrator and support engineer before becoming an Oracle DBA He is responsible for maintaining the Oracle RAC Assurance Support Team’s Starter Kit and Best Practices documentation He has reviewed many Oracle RAC configurations and provided feedback for best practices configuration He is also actively supporting Oracle customers with Oracle RAC-related issues This page intentionally left blank Oracle Database 11g Release High Availability Maximize Your Availability with Grid Infrastructure, Oracle Real Application Clusters, and Oracle Data Guard, Second Edition Scott Jesse Bill Burton Bryan Vongray New York  Chicago  San Francisco  Lisbon  London  Madrid  Mexico City  Milan  New Delhi  San Juan  Seoul  Singapore  Sydney  Toronto Copyright © 2011 by The McGraw-Hill Companies, Inc (Publisher) All rights reserved Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher ISBN: 978-0-07-175207-7 MHID: 0-07-175207-2 The material in this eBook also appears in the print version of this title: ISBN: 978-0-07-175208-4, MHID: 0-07-175208-0 All trademarks are trademarks of their respective owners Rather than put a trademark symbol after every occurrence of a trademarked name, we use names in an editorial fashion only, and to the benefit of the trademark owner, with no intention of infringement of the trademark Where such designations appear in this book, they have been printed with initial caps McGraw-Hill eBooks are available at special quantity discounts to use as premiums and sales promotions, or for use in corporate training programs To contact a representative please e-mail us at bulksales@mcgraw-hill.com Oracle is a registered trademark of Oracle Corporation and/or its affiliates All other trademarks are the property of their respective owners, and McGraw-Hill makes no claim of ownership by the mention of products that contain these marks Information has been obtained by Publisher from sources believed to be reliable However, because of the possibility of human or mechanical error by our sources, Publisher, or others, Publisher does not guarantee to the accuracy, adequacy, or completeness of any information included in this work and is not responsible for any errors or omissions or the results obtained from the use of such information Oracle Corporation does not make any representations or warranties as to the accuracy, adequacy, or completeness of any information contained in this Work, and is not responsible for any errors or omissions TERMS OF USE This is a copyrighted work and The McGraw-Hill Companies, Inc (“McGrawHill”) and its licensors reserve all rights in and to the work Use of this work is subject to these terms Except as permitted under the Copyright Act of 1976 and the right to store and retrieve one copy of the work, you may not decompile, disassemble, reverse engineer, reproduce, modify, create derivative works based upon, transmit, distribute, disseminate, sell, publish or sublicense the work or any part of it without McGraw-Hill’s prior consent You may use the work for your own noncommercial and personal use; any other use of the work is strictly prohibited Your right to use the work may be terminated if you fail to comply with these terms THE WORK IS PROVIDED “AS IS.” McGRAW-HILL AND ITS LICENSORS MAKE NO GUARANTEES OR WARRANTIES AS TO THE ACCURACY, ADEQUACY OR COMPLETENESS OF OR RESULTS TO BE OBTAINED FROM USING THE WORK, INCLUDING ANY INFORMATION THAT CAN BE ACCESSED THROUGH THE WORK VIA HYPERLINK OR OTHERWISE, AND EXPRESSLY DISCLAIM ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE McGraw-Hill and its licensors not warrant or guarantee that the functions contained in the work will meet your requirements or that its operation will be uninterrupted or error free Neither McGraw-Hill nor its licensors shall be liable to you or anyone else for any inaccuracy, error or omission, regardless of cause, in the work or for any damages resulting therefrom McGraw-Hill has no responsibility for the content of any information accessed through the work Under no circumstances shall McGraw-Hill and/or its licensors be liable for any indirect, incidental, special, punitive, consequential or similar damages that result from the use of or inability to use the work, even if any of them has been advised of the possibility of such damages This limitation of liability shall apply to any claim or cause whatsoever whether such claim or cause arises in contract, tort or otherwise FREE SUBSCRIPTION TO ORACLE MAGAZINE GET YOUR Oracle Magazine is essential gear for today’s information technology professionals Stay informed and increase your productivity with every issue of Oracle Magazine Inside each free bimonthly issue you’ll get: If there are other Oracle users at your location who would like to receive their own subscription to Oracle Magazine, please photocopy this form and pass it along Three easy ways to subscribe: Web oracle.com/oraclemagazine Fax +1.847.763.9638 Mail P.O Box 1263, Skokie, IL 60076-8263 Copyright © 2008, Oracle and/or its affiliates All rights reserved Oracle is a registered trademark of Oracle Corporation and/or its affiliates Other names may be trademarks of their respective owners Index  International Organization for Standardization See ISO interprocess communications (IPC), 73 IP addresses dynamic, 59 failovers, 61 SCAN VIP, 56, 59, 82, 268–269 static, 56, 57 virtual, 61, 81–82, 103, 267–268 IP network multipathing (IPMP), 190 IPC (interprocess communications), 73 IPC traffic, 189–190 IPD/OS See Cluster Health Monitor IPMI (Integrating Intelligent Platform Management Interface), 66, 105 IPMP (IP network multipathing), 190 iptables, 28 iSCSI, 42–44 ISO files, 29 ISO (International Organization for Standardization) images, 23 iso_pool directory, 23 J JDBC clients, 251 JDBC connections, 440, 441 job role separation, 79 job system, 445 jumbo frames, 223–224 K kernel parameters, 84 kill process, 66 Knowledge Base searches, 489 L LCK (lock) process, 204 leases, node number, 64–65 LGWR attribute, 306 library cache, 208 libskgxn.so file, 49 Linux systems examples, xxxv HugePages feature, 225 preparing Linux/Oracle RAC clusters, 85–97 Red Hat Package Manager, 448 listener administration, 484 listener registration, 266–267 listeners in clusters, 264–269 local, 248–249, 266 SCAN See SCAN listeners live nodes, LM Heartbeat Monitor (LMHB), 206 LMD (lock manager daemon) process, 204 LMHB (LM Heartbeat Monitor), 206 LMON (lock monitor) process, 205 LMS (lock manager server) process, 205 LNS (log-write network-server) process, 306 load balancing See also workloads client-side, 248, 271 configuration, 249 considerations, 190 goals, 249 performance and, 248–249 Remote Load Balancing, 250 SCAN listeners and, 248 server-side, 248–249 load balancing advisory, 251–252 load testing, 418–419 local disks, 78 local GI home directory, 78 local listeners, 248–249, 266, 268 local VIPs, 268 LOCAL_LISTENER parameter, 266 LOCATION attribute, 305 lock manager daemon (LMD) process, 204 lock manager server (LMS) process, 205 lock monitor (LMON) process, 205 lock (LCK) process, 204 501 502   Oracle Database 11g Release High Availability log files agent logs, 66 archivelogs See archivelogs cluster alert logs, 74–75 error logs, 250 event logs, 250 Flashback logs, 414–416 inaccessible, 436 RDBMS alert logs, 74 redo logs See redo logs retention period, 328–329 rootcrs log, 114–115 undo logs, 192 log transport compression, 307 log transport services, 305–309 logical errors, 10 logical standby databases converting physical standbys to, creating, 300–305 deleting archived logs, 328–329 error recovery, 330–331 managing, 326–332 monitoring, 327–328 overview, 286–287 protecting from user changes, 329–330 supported/unsupported objects, 300–302 switchover to, 332–333 Logical Unit Numbers See LUNs LogMiner dictionary, 303, 304, 402, 408, 409 log-write network-server (LNS) process, 306 lsdsk command, 128–129 lsnrctl commands, 194 LUN alignment, 92 LUN persistence, 93 LunarTrax data corruption, 380, 390 GNS and, 99 overview, 4–5 performance issues, 284 planning for expansion, 11–13 LUNs (Logical Unit Numbers), 92 M MA (Maximum Availability) considerations, xxvii–xxviii, described, xxvii, xxviii, 311–312 Oracle’s integrated approach to, xxix–xxx overview, xxviii–xxxi MAA (Maximum Availability Architecture) backup strategies, 358–371 cost control, xxix defined, xxviii manageability of, xxix MAA Advisor, 486 MAA Console, 485–486 MAA environment administration, 482–489 MAA workshops adding shared disks/NICs, 36–39 applying patches, 230 backing up to tape from standby, 388–389 changing protection mode, 312–313 compressed backup sets, 360–361 configuring Flash Recovery Area, 349–350 configuring/testing TAF, 273–274 converting physical to snapshot standby, 299–300 creating ACFS for DB home, 149–153 creating disk groups, 137–142 creating general purpose ACSF, 155–158 creating logical standby database, 302–305 creating physical standby database, 288–298 creating simple FAN callout, 263 creating VMs from templates, 40–42 creating VMs with Oracle VM Manager, 30–36 Index  Data Recovery Advisor, 380–382 deleting nodes from clusters, 118–119 enabling Flashback Database, 415 failover to physical standby, 337–338 implementing recovery catalogs, 357 installing management agents, 471–473 installing Oracle Grid Control, 461–471 installing Oracle VM Manager, 26–28 opening standby in read-only mode, 322–326 Oracle Data Guard Broker configuration, 425–427 Oracle Data Guard protection level, 430 Oracle RAC database creation, 185–190 overview, xxxiv–xxxv performing Flashback Database, 417–418 policy-managed databases, 237–238 preparing Linux/Oracle RAC clusters, 85–97 RDBMS software install, 180–182 recovering from complete loss, 373–375 reinstating broker-managed database, 435 reinstating database after failover, 339–340 setting ASM disk strings, 128–129 setting persistent configuration parameters, 353–354 setting up ACFS replication, 163–165 switchover to physical standby, 333–335 using Flashback Transaction, 410–413 using incrementally updated backups, 364–366 WebLogic for use by Grid Control, 453–460 MAIL_FAILURE attribute, 307 maintenance importance of, 6–7 patches, 231 planned, 5, 6–8 proactive strategies, 231 reducing impact of, 7–8 Manageability Monitor (MMON) process, 218 Management Agents, 471–473 administration tasks, 475–476 installing via push method, 471–473 metrics gathering, 479 obtaining status of, 476 overview, 446, 471 setting up, 468 starting/stopping, 476 management domain, 21 Management Repository, 446 Maximum Availability See MA Maximum Availability Architecture See MAA MDNSD (Multicast Domain Name Services daemon), 71 mean time to recovery (MTTR), 362 media management layer (MML), 382–385 media recovery, 210–211, 226 Media Recovery Process (MRP), 315, 317–318 media sensing, 82 memory, 22, 83, 205, 225, 346–347 memory starvation, 83 metadata, ASM, 143 metric settings, 479–480 503 504   Oracle Database 11g Release High Availability Middleware Home directory, 454, 456, 464, 473 mirror resynchronization, 134, 201 mirrored extent sets, 200 mirroring, 50, 130–132, 200 mkfs command, 157 MML (media management layer), 382–385 MMON (Manageability Monitor) process, 218 MOS (My Oracle Support), 445, 488–489 MRP (Media Recovery Process), 315, 317–318 MTTR (mean time to recovery), 362 Multicast Domain Name Services daemon (MDNSD), 71 multiplexing, 384–385 My Oracle Support (MOS), 445, 488–489 N NAS (Network Attached Storage), 78, 79 natural disasters, 13 Net Configuration Assistant (NETCA), 253 NETCA (Net Configuration Assistant), 253 NET_TIMEOUT attribute, 307 Network Attached Storage (NAS), 78, 79 Network File System See NFS network interface bonding, 82 network interface cards See NICs network time protocol (ntp), 61–62 Network Time Protocol daemon (NTPD), 87 networks See also connections disconnects, 307, 313–314, 317 IPC traffic, 189–190 NFS (Network File System), 78 NFS mounts, 79, 81, 226–227 NIC bonding, 123 NICs (network interface cards), 36–39 node applications, 259–261 node kill, 66 node number pinning/leases, 64–65 node numbers, 64–65 nodes See also clusters adding to clusters, 116–117 assigning numbers to, 65 communication between, 49–50 deleting from clusters, 117–119 evicting from clusters, 66 extending Oracle RAC databases to, 191–193 hung, 66 leases, 64–65 missing heartbeats, 49 pinned/unpinned, 64–65, 195 state of, 49 status of, 65 notifications e-mail, 180, 446, 454, 480, 482 events, 249–253 Fast Application Notification, 251, 278, 440–442 methods for, 480–482 Oracle Grid Control, 480–482 Oracle Notification Service, 73, 250–252, 278 rules, 481 subscribing to, 481–482 nslookup command, 58 ntp (network time protocol), 61–62 NTPD (Network Time Protocol daemon), 87 O objects See database objects the Observer, 436, 438–439 OCFS (Oracle Cluster File System), 81, 227–228 OCR (Oracle Cluster Registry), 53–55, 233, 234–235 Index  OCR files, 53–55, 134 ocrcheck command, 53–54, 55 ocrconfig command, 54 OEM (Oracle Enterprise Manager) considerations, 175 creating services, 253–257 event monitoring, 214–215 Grid Control See Oracle Grid Control services integration, 278–279 OHASD (Oracle HA Services Daemon), 67, 68–73, 110 OHASD oraagent, 71–72 OHASD root agent, 72 OID (Oracle Internet Directory), 309 oifcfg command, 123 oinstall group, 175 OLR (Oracle Local Registry), 55–56 olsnodes command, 64, 65 OMF (Oracle Managed Files), 192, 351 OMS (Oracle Management Service) administration tasks, 474–475 creating OMS users, 475 described, 446 obtaining status of, 475 starting, 474 stopping, 474–475 OMS Home directory, 464, 473 OMS Server, 470, 474–475, 476, 479 online patching, 175, 231 ONS (Oracle Notification Service), 73, 250–252, 278 OPatch utility, 175, 229 Openfiler, 43 oper group, 176 operating system packages, 84 OPS (Oracle Parallel Server), xxxi, 48 oraagent, 66, 71–72, 73 Oracle Active Data Guard, 286, 287, 322–326, 441 Oracle Automatic Storage Management See ASM Oracle base directory, 91, 106 Oracle Cluster Registry See OCR Oracle Clusterware See also clusterware advantages of, 12 ASM and, 233 considerations, 432 described, disk groups, 183 third-party clusterware and, 49 Oracle Data Guard, 283–340 architecture overview, 424 combining with Oracle RAC, configuration, 425–430, 436–439 considerations, 13–14 cross-platform configurations, 285 described, 424 DGMGRL interface, 424–425, 430 failovers See Fast-Start Failover FAN and, 278 gap detection/resolution, 314–316 in geographically separate locations, 13 log transport services, 305–309 managing with Grid Control, 486–488 OEM interface, 424–425 Oracle Active Data Guard, 286, 287, 322–326, 441 protection level, 430 Oracle Data Guard Broker, 423–442 configuration, 425–430, 436–439 monitoring, 427–429 network disconnects, 313–314 Oracle Active Data Guard, 322–326 Oracle Grid Control, 487 Oracle RAC/standby/replication environments, 275–278 overview, 284–285 performing failovers, 433–434 performing switchovers, 431–433 planned maintenance and, protection modes, 310–313 505 506   Oracle Database 11g Release High Availability Oracle Data Guard Broker (cont.) reinstating broker-managed database, 435 relationship with RMAN, 385–391 role transitions, 332–340, 431–435 switchovers, 332–335 Oracle Database considerations, 64 install options, 174–176 installing earlier releases of, 194–198 software install, 180–182 Oracle Database 10g, xxxi, 48, 49 Oracle Database 10g Release 1, 121 Oracle Database 10g Release 2, 121 Oracle Database 11g, xxx, xxxi, 48, 49 Oracle Database 11g Release 1, xxxi, 49, 121 Oracle Database 11g Release 2, xxix, 17, 48 Oracle Database Grid See also GI architecting, 3–17 future growth of, 11–13 infrastructure See GI maintaining See maintenance planning for expansion, 11–13 testing See testing Oracle Easy*Connection, 275 Oracle Enterprise Manager See OEM Oracle GoldenGate, Oracle Grid Control, 443–489 See also GI entries administration of, 473–478 ADR integration, 489 architecture, 446 creating Super Administrator, 475 hardware requirements, 448 installing, 461–471 kernel parameters, 450 key features, 445 Knowledge Base searches, 489 MAA environment administration, 482–489 Management Agents See Management Agents managing ACFS, 484 managing ASM, 484 managing clusters, 483 managing database backups, 488 managing disk groups, 484 managing grid infrastructure, 483–484 managing Oracle Data Guard, 486–488 managing Oracle RAC databases, 485–486 managing Oracle VM, 483–484 managing targets, 476–478 metric settings, 479–480 monitoring with, 445, 478–480 My Oracle Support, 445, 488–489 notifications, 480–482 Oracle Management Repository, 446 Oracle Management Service, 446 OS packages, 448–449 OS users/groups, 450 overview, 444–445 patches, 458–460, 462, 484, 488–489 plug-ins, 445 preparing to install, 447–460 prerequisites, 447–451, 464 reports, 445 repository database, 451–453 Smart Update utility, 453, 458–460 supported platforms, 447 temp space, 450–451 Oracle grid infrastructure See GI Oracle Grid Manager, 468 Oracle HA Services Daemon See OHASD Oracle home directory, 106 Oracle Internet Directory (OID), 309 Oracle Kernel Services, 144 Oracle Local Registry (OLR), 55–56 Index  Oracle Managed Files (OMF), 192, 351 Oracle Management Agents See Management Agents Oracle Management Repository, 446 Oracle Management Server, 470, 474–475, 476, 479 Oracle Management Service See OMS Oracle Notification Service (ONS), 73, 250–252, 278 Oracle ORION (Oracle IO Numbers), 218 Oracle Parallel Server (OPS), xxxi, 48 Oracle RAC extended distance clusters, 198–201 home directory, 192 instances, 204–209, 249, 259, 432 metrics, 221–222 Oracle RAC (Real Application Clusters), 171–280 administration See Oracle RAC administration ADR in, 240–241 combining with Oracle Data Guard, configuring, 198–201 considerations, 48 described, enabling archiving, 190–191 file system options, 176–180 install options, 174–176 installing, 180–190 local vs extended, 199–200 networking requirements, 81–82 parameters, 189–190 planned maintenance and, redundant database access, relationship with RMAN, 385–391 shared storage, 199–201 standby databases See standby databases supported VM configurations for, 44 templates, 44–45 workload management, 190–191 Oracle RAC administration, 203–241 Automatic Database Diagnostic Monitor, 218, 220–222 Automatic Diagnostic Repository, 74, 239–241 Automatic Workload Repository, 218–222 cache coherency, 208–209 Cache Fusion, 206–207, 208 Cluster Health Monitor, 123, 217 diagnostic data, 239–241 dynamic resource mastering, 207 management via policies, 237–239 managing diagnostic data, 239–241 media recovery, 210–211 monitoring tools, 213–219 Oracle IO Numbers, 218 Oracle RAC vs single-instance, 204–209 OS Watcher, 215–217 patches See patches problem resolution, 241 reconfiguration, 207 redo/rollback operations, 209–211 SRVCTL utility See SRVCTL utility Oracle RAC databases adding instances to, 192–193 deleting instances from, 193 extending to new nodes, 191–193 managing with Grid Control, 485–486 Oracle RAC environments archiving in, 229 cache coherency in, 208–209 Flash Recovery Area and, 394–395 instance recovery, 209–210 interconnect performance, 222–224 507 508   Oracle Database 11g Release High Availability Oracle RAC environments (cont.) monitoring, 213–218 parallelism in, 211–213 redo/rollback operations, 209–211 snapshot controlfile in, 391–392 TCP timeouts, 267–268 vs single-instance environment, 204–209 Oracle RDBMS See Oracle Database Oracle Recovery Manager See RMAN Oracle Secure Backup (OSB), 382–384, 385 Oracle Smart Update utility, 453, 458–460 Oracle stack, xxix–xxx, 17, 176 Oracle Streams, Oracle Technology Network (OTN), 123, 218 Oracle Universal Installer (OUI), 85, 114, 175, 180–182 Oracle VM, 19–45 See also virtual machines adding shared disks/NICs, 36–39 considerations, xxxv creating VMs, 30–36 creating VMs from templates, 40–42 managing See Oracle VM Manager managing via Grid Control, 483–484 overview, 20–21 supported configurations, 44 virtualization basics, 20 Oracle VM Manager, 29–44 accessing, 30 described, 25 hardware requirements, 25 installing, 26–28 resources, 29 server pools, 29 servers, 29 virtual machines, 29–39 Oracle VM Server, 21–25 hardware requirements, 21–22 installing, 22–25 multiple servers, 42–44 using, 39–42 Oracle VM templates, 44–45 Oracle WebLogic Server, 447, 453–460, 465–466 oracleasm module, 94–95, 115, 144 ora.ctssd process, 72 ora.drivers.acfs resource, 72 orarootagent, 66, 73 ORION (Oracle IO Numbers), 218 OS file copies, OS groups, 175–176, 450 OS Watcher (OSW), 215–217 OSB (Oracle Secure Backup), 382–384, 385 OSW (OS Watcher), 215–217 OTN (Oracle Technology Network), 123, 218 OUI (Oracle Universal Installer), 85, 114, 175, 180–182 outages, 4, 398, 419, 420, 478, 486 OVM See Oracle VM ovs-agent directory, 23 P Parallel Automatic Tuning, 212 parallel execution (PX) servers, 212 parallel queries, 212–213, 224 PARALLEL_EXECUTION_MESSAGE_ SIZE parameter, 189 parallelism, 211–213 parallelization, 384–385 PARALLEL_MAX_SERVERS parameter, 189 paravirtualized guests, 21 partitioning, 213, 228 partner disks, 131 password file, 291–292, 295, 309, 319 passwords authentication, 309 errors, 295 Index  forgotten, 27 Oracle RAC setup, 197 SYS/SYSTEM, 186, 187 patch plans, 488–489 patches, 229–232 considerations, 183 critical, 232 described, 229 Grid Control, 458–460, 462, 484, 488–489 hot, 175, 231 online, 175, 231 OPatch utility, 175, 229 Oracle RAC, 7, 183 recommended, 232, 488 rolling updates, 229–230 patchset releases, 123–124 patchset updates (PSUs), 6, 183, 232 patchsets, 6, 229–232 performance ASM, 484 block change tracking, 366 considerations, xxix interconnect, 222–224 load balancing and, 248–249 maximum, 312–313 partitioning and, 213 tuning, 218–225 persistent configuration parameters, 351–354 PGA (Program Global Area), 346 physical memory, 83 physical standby databases changes to primary and, 318–320 converting to logical standby databases, creating, 288–298 creating backups with, 387–389 failover to, 337–338 managing, 316–326 monitoring, 320–321 Oracle Data Guard Broker configuration, 425–427 overview, 285–286 read-only mode, 322–326 starting, 316–317 switchover to, 332–335 synchronizing, 325 PING process, 206 pinning, node number, 64–65 pinning/unpinning nodes, 64–65, 195 planned maintenance, 5, 6–8 See also maintenance PMON (Process Monitor) discovery, 266–267 point-in-time recovery, 395, 398, 405, 413 policy-based cluster management, 16–18, 62–64 policy-managed databases adding instances to, 192 considerations, 237 converting database to, 237–238 deleting instances from, 193 services in, 247, 262 special options, 262 port range, 469 PRE_PAGE_SGA parameter, 189 preparer process, 326 preserved snapshot set, 220 primary database described, 284 physical changes on, 318–320 reinstating as standby after failover, 339–340 private drives, 174–175 problem resolution, 241 Process Monitor (PMON) discovery, 266–267 processors, 82–83 production systems, 16 Program Global Area (PGA), 346 protection modes, 310–313 PSUs (patchset updates), 6, 183, 232 publish_pool directory, 23 push installation method, 471–473 PX (parallel execution) servers, 212 509 510   Oracle Database 11g Release High Availability Q queries Flashback Query, 11, 400–403 Flashback Transaction Query, 404–405 Flashback Versions Query, 403–404 parallel, 212–213, 224 previous point in time, 402 on read-only data, 322–323 quorum failure group, 201 R RAC See Oracle RAC rapid connect-time failover, 61 raw devices, 80, 81, 226–227 RDBMS See Oracle Database RDBMS alert logs, 74 RDBMS home, 79, 80, 84, 174–175 RDBMS servers, 64 RDS (Reliable Datagram Sockets) protocol, 223, 224 read failure groups, 134 reader farms, 286, 324 reader process, 326 read-only mode, 322–326 Real-Time Apply feature, 317 real-time query See Oracle Active Data Guard rebalancing operations, 134–135 reconfiguration, 207 recovery See also backups; disaster planning block meda recovery, 375–376 from complete loss, 373–375 considerations, 395–396 controlfiles, 373–375 crash, 189 Data Recovery Advisor, 376–382 datafiles, 373, 375 Flashback Recovery, 397–420 full database, 372 Health Monitor, 377–380 instance, 209–210 media, 210–211, 226 overview, 371 performing, 371–382 point-in-time, 395, 398, 405, 413 restore/recover operations, 372–375 starting/stopping for standby, 317–318 tablespace, 373 tablespace-point-in-time, 398 testing recovery scenarios, 16 transaction, 189 with updated datafile copies, 375 user errors, 10–11 recovery catalog, 344, 356–357 Recovery Manager See RMAN Recovery Time Objective (RTO), 287 recovery window, 368 Recycle Bin, 406–408 Red Hat Package Manager See RPM redo log kick, 229 Redo Logfile Size Advisor, 189 redo logs, 209–211 adding/dropping, 319 considerations, 192, 209 instance recovery, 209–210 media recovery, 210–211 size, 189 standby redo logs, 292–293 redo threats, 211 redo/rollback operations, 209–211 redundancy ASM, 184, 200–201 considerations, 8–9 disk groups, 50, 184, 200–201 external, 50 high, 50 normal, 50 SCAN listeners, 268–269 testing for, 15 redundancy policies, 368 Redundant Interconnect feature, 190 Index  Reliable Datagram Sockets (RDS) protocol, 223, 224 RELY constraint, 302 remote file server (RFS) process, 309–310 Remote Load Balancing (RLB), 250 Remote Slave Monitor (RSMN) process, 206 REMOTE_LISTENER parameter, 249 REOPEN attribute, 307 replace command, 54 replication, 161–165 replication environment, 275–278 reports ADDM, 221 AWR, 219 custom, 445 Grid Control, 445 obsolete backups/copies, 368–369 read-only mode, 322–326 repository database configuring, 452–453 creating, 451–452 Resource Manager, 246–247 resources local, 53 managed by OCR, 53 managing, 65–66, 246–247 server pools, 63 used by Oracle VM Manager, 29 restores, testing, 15–16 retention policies, 367–370 RETRIES parameter, 274, 276 return on investment (ROI), 287 RFS (remote file server) process, 309–310 RLB (Remote Load Balancing), 250 RMAN (Recovery Manager) active database duplication, 387 archivelogs, 345–346 backup sets, 358–361 backup strategies, 348–357 backup-based duplication, 386 basics, 344–348 block meda recovery, 375–376 building standby database, 385–387 cloning database, 385–387 command-line usage, 347–348 configuration parameters, 351–354 configuring clusters, 391–396 considerations, 343 controlfile, 345–346 data blocks, 346–347 database duplication, 294–297 datafiles, 361–362 described, 9, 344 FRA See Flash Recovery Area Oracle Data Guard and, 385–391 Oracle RAC and, 391–396 recovery catalog, 356–357 standby database instantiation, 289–290 RMAN executable, 344 RMAN-20242 errors, 393, 394 ROI (return on investment), 287 role transitions Oracle Data Guard Broker, 431–435 via failovers, 332–340 role-based services, 440–442 role-separated management, 64 rollback/redo operations, 209–211 rolling updates, 229–230 rolling upgrades, 175 root scripts, 108–117, 121 rootcrs log, 114–115 root.sh script, 109–116, 181, 192 rootupgrade.sh script, 119, 121, 124 rows, uniquely identifying, 301–302 RPM (Red Hat Package Manager), 448 RPM files, 28, 94 RSMN (Remote Slave Monitor) process, 206 RTO (Recovery Time Objective), 287 runInstaller command, 196 511 512   Oracle Database 11g Release High Availability runlevel, 69 running_pool directory, 24, 43 S sabotage, 13 SAME philosophy, 130–132 SAN (Storage Area Network), 199 SAN buffer credits, 199 SAN networks, 199 sandbox environments, 16 SBT (System Backup to Tape), 365, 386, 387 SCAN (Single Client Access Name), 58–61, 248 SCAN listeners administration, 484 considerations, 268–269 load balancing and, 248 overview, 264–265 redundancy, 268–269 using, 58–61 SCAN name, 56, 82 SCAN VIP addresses, 56, 59, 82, 268–269 SCNs (system change numbers), 327, 347 script agent, 66, 74 scripts fixup, 107–108, 182 root, 108–117, 121 root.sh, 109–116, 181, 192 rootupgrade.sh, 119, 121, 124 scsi-id, 95–96 Secure Shell See SSH Secure Sockets Layer (SSL), 28, 291, 309, 468 security ASM Cluster File System, 165–167 encryption, 168–169, 320 log transport services, 309 passwords See passwords security identifiers (SIDs), 187, 297 seed_pool directory, 24 sequence caches, 224–225 sequences, 224–225 Server Control Utility See SRVCTL utility server parameter files (SPFILEs), 80 server pools attributes, 237 described, 12–13, 62 free pool, 64 generic pool, 64 minimum/maximum servers in, 62, 63 Oracle VM Manager and, 29 policy-based clusters and, 62–64 resources, 63 rules/policies, 62–64 subpools, 63 servers described, 29 DHCP, 59 DNS, 57, 59, 81, 248 number of, 12–13, 237 older, 13 OMS Server, 470, 474–475, 476, 479 Oracle Parallel Server, xxxi, 48 Oracle VM See Oracle VM Server Oracle VM Manager and, 29 parallel execution, 212 RDBMS, 64 WebLogic Server, 447, 453–460, 465–466 server-side load balancing, 248–249 server-side TAF, 270 SERVICE attribute, 305 service level agreements (SLAs), 213 service request (SR) management, 489 service time, 252 service-level goals, 252 SERVICE_NAMES parameter, 245 services, 243–280 applications as, 243–280 basic concepts, 244–247 Index  creating, 253–257 creating with SRVCTL, 261–262 from database perspective, 245–247 enabled/disabled, 245 integration with OEM, 278–279 isolating, 245 managing with SRVCTL, 259, 261–262 modifying, 259 in policy-managed databases, 247, 262 prioritizing, 246 relocating, 259 Resource Manager and, 246–247 role-based, 440–442 singleton, 247 uniform, 247 viewing in database, 258–259 as workloads, 244 shared disks adding with Oracle VM Manager, 36–39 multiple VM servers, 42–44 resources, 29 sharedDisk directory, 24–25 using iSCSI for, 42–44 shared everything architecture, 7, 80 shared home directories, 78, 174–175 shared nothing architecture, 80 shared storage for clusterware files, 79–81 Oracle RAC, 199–201 virtual disks, 29 sharedDisk directory, 24–25 show parameter command, 258 SID prefix, 187 SIDs (security identifiers), 187, 297 Single Client Access Name See SCAN single-instance environment, 204–209 SLAs (service level agreements), 213 Smart Update utility, 453, 458–460 snapshot controlfile, 355, 391–392 snapshot standby databases creating, 298–300 overview, 286, 298 snapshots ACFS read-only, 159–160 AWR reports, 219–220 software failures, obsolete, xxviii “perfect” vs “imperfect,” xxviii updates, 122–123 vaporware, xxviii SPFILEs (server parameter files), 80 SQL Apply engine changing default behavior of, 331–332 error recovery, 330–331 monitoring progress, 327–328 overview, 326–327 starting/stopping, 327 SR (service request) management, 489 SRVCTL utility, 233–237 adding/removing databases, 235 creating services, 253, 261–262 described, 233 managing ASM, 233–234 managing databases, 234–235 managing instances, 234–235 managing node applications, 259–261 managing services, 259, 261–262 SSH (Secure Shell), 86, 87 SSH connectivity, 100–101, 102 SSL (Secure Sockets Layer), 28, 291, 309, 468 standby databases See also Oracle Data Guard backing up to tape, 388–389 building with RMAN, 385–387 logical See logical standby databases overview, 284–285 physical See physical standby databases 513 514   Oracle Database 11g Release High Availability standby databases See also Oracle Data Guard (cont.) reinstating, 419 snapshot See snapshot standby databases starting/stopping recovery, 317–318 standby environments, 275–278 standby redo logs, 292–293 Standby Statspack, 325 standby_file_management parameter, 319 start argument, 69 start service command, 261 startup mount command, 316 static IP addresses, 56, 57 stop argument, 69 stop command, 70 Storage Area Network See SAN storage options ASM See ASM database files, 176–180 NAS, 78, 79 shared, 7, 79–81 striping, 130–132 swap space, 83 switchovers, 332–335, 431–433 SYNC attributes, 306, 311, 312 sysman schema, 452 System Backup to Tape (SBT), 365, 386, 387 system change numbers (SCNs), 327, 347 system clock, 62 System.img file, 24 T tablespace-point-in-time recovery (TSPITR), 398 tablespaces dropping, 319 Oracle Grid Manager, 468 recovering, 373 transportable, 319 TAF (Transparent Application Failover), 251, 270–276 tagging, ACFS, 160–161 tape, backups to, 365, 382–389 target databases, 296, 344 target discovery, 477 targets ASM, 476–477 Grid Control, 476–478 TCP timeouts, 267–268, 313–314 TDE (Transparent Data Encryption), 320 templates creating virtual machines from, 40–42 directory for, 24 disk group, 135–136 obtaining, 45 Oracle RAC, 44–45 Oracle VM, 44–45 virtual machines, 29 terrorist attacks, 13 test plans, 16 testing, 14–18 backups, 15–16 disaster recovery scenario, 16 load testing, 418–419 overview, 14–15 production systems and, 16 redundancy, 15 restores, 15–16 sandbox environments and, 16 Transparent Application Failover, 273–274 throughput, 252 timeouts, 267–268, 307, 313–314 TNS (Transparent Network Substrate) aliases, 290 trace files, 74–76 transaction recovery, 189 transactional changes, 408–413 Transparent Application Failover (TAF), 251, 270–276 Transparent Data Encryption (TDE), 320 Index  Transparent Network Substrate (TNS) aliases, 290 troubleshooting, 114–115, 402, 439, 489 TSPITR (tablespace-point-in-time recovery), 398 tuning, 218–225 U UDP (User Datagram Protocol) buffers, 222–223 UID (Universal ID), 52 undo guarantee attribute, 401–402 undo logs, 192 undo management, 400–402 undo retention, 401–402 undo_retention parameter, 401 Universal ID (UID), 52 updates, 122–123 upgrading ASM, 120–122 to GI, 119–122 User Datagram Protocol (UDP) buffers, 222–223 user errors See also errors considerations, xxviii–xxxix planning for, protecting against, 10–11 recovering from, 10–11 utility computing, 243–280 V v$ views, 321 VALID_FOR attribute, 308–309 vaporware, xxviii VIP (virtual IP) addresses, 61, 81–82, 103, 267–268 VIPs, SCAN, 56, 59, 82, 268–269 virtual disks, 29 virtual IP (VIP) addresses, 61, 81–82, 103, 267–268 virtual machines See also Oracle VM creating from templates, 40–42 creating with Oracle VM Manager, 29–36 described, 29 system images, 29 templates, 29 virtualization, 20 See also Oracle VM VM Manager See Oracle VM Manager VM Server See Oracle VM Server vm.cfg file, 24 volumes, dynamic, 80, 143–148 voting disks, 49–52 on extended clusters, 201 number of, 104 overview, 49–50 tie-breaking, 201 voting files, 49–52, 134 W wait events, 221 Wavelength Division Multiplexing (WDM), 199 WDJ7patch, 458–460 WDM (Wavelength Division Multiplexing), 199 WebLogic Server, 447, 453–460, 465–466 Workload Repository, 219–221 See also AWR workloads balancing See load balancing comparing, 219–220 managing, 190 monitoring, 218 services as, 244 workshops See MAA workshops 515 ... 40 125 128 137 144 149 155 163 ix Oracle- Regular / Oracle Database 11g Release High Availability / Jesse et al / 20 8-0 x    Oracle Database 11g Release High Availability part iI Oracle Real Application. .. PRODUCTS ARE IN USE AT YOUR SITE? (check all that apply) ORACLE DATABASE o 01 Oracle Database 11g o 02 Oracle Database 10 g o 03 Oracle9 i Database o 04 Oracle Embedded Database (Oracle Lite, Times... Oracle customers with Oracle RAC-related issues This page intentionally left blank Oracle Database 11g Release High Availability Maximize Your Availability with Grid Infrastructure, Oracle Real