Page iii Unix Backup and Recovery W. Curtis Preston Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • Tokyo Page iv Disclaimer: This netLibrary eBook does not include data from the CD-ROM that was part of the original hard copy book. Unix Backup and Recovery by W. Curtis Preston Copyright (c) 1999 O'Reilly & Associates, Inc. All rights reserved. Printed in the United States of America. Published by O'Reilly & Associates, Inc., 101 Morris Street, Sebastopol, CA 95472. Editor: Gigi Estabrook Production Editor: Clairemarie Fisher O'Leary Printing History: November 1999: First Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. The association between the image of an Indian gavial and the topic of Unix backup and recovery is a trademark of O'Reilly & Associates, Inc. While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. This book is printed on acid-free paper with 85% recycled content, 15% post-consumer waste. O'Reilly & Associates is committed to using paper with the highest recycled content available consistent with high quality. ISBN: 1-56592-642-0 Page v This book is dedicated to my lovely wife Celynn, my beautiful daughters Nina and Marissa, and to God, for continuing to bless my life with gifts such as these. -W. Curtis Preston Page vii TABLE OF CONTENTS Preface xiii I. Introduction 1 1. Preparing for the Worst 3 My Dad Was Right 3 Developing a Disaster Recovery Plan 4 Step 1: Define (Un)acceptable Loss 5 Step 2: Back Up Everything 7 Step 3: Organize Everything 10 Step 4: Protect Against Disasters 13 Step 5: Document What You Have Done 15 Step 6: Test, Test, Test 16 Step 6: Test, Test, Test 16 Put It All Together 17 2. Backing It All Up 18 Don't Skip This Chapter! 18 Why Should You Read This Book? 19 How Serious Is Your Company About Backups? 22 You Can Find a Balance 25 Deciding What to Back Up 30 Deciding When to Back Up 38 Deciding How to Back Up 43 Storing Your Backups 52 Testing Your Backups 56 Monitoring Your Backups 58 Page viii Following Proper Development Procedures 59 Unrelated Miscellanea 60 Good Luck 65 II. Freely Available Filesystem Backup & Recovery Utilities 67 3. Native Backup & Recovery Utilities 69 An Overview 69 Backing Up with the dump Utility 73 Restoring with the r estore Utility 91 Restoring with the r estore Utility 91 Limitations of dump and restore 101 Features to Check For 102 Backing Up and Restoring with the cpio Utility 103 Backing Up and Restoring with the tar Utility 114 Backing Up and Restoring with the dd Utility 122 Comparing tar, cpio, and dump 127 How Do I Read This Volume? 129 4. Free Backup Utilities 141 The hostdump.sh Utility 141 The infback.sh, oraback.sh, and syback.sh Utilities 142 A Really Fast tar Utility: star 142 Recording Configuration Data: The SysAudit Utility 143 Displaying Host Information: The SysInfo Utility 144 Performing Remote Detections: The queso Utility 144 Mapping Your Network: The nmap Utility 145 AMANDA 146 III. Commercial Filesystem Backup & Recovery Utilities 185 5. Commercial Backup Utilities 187 What to Look For 188 Full Support of Your Platforms 189 Backup of Raw Partitions 191 Backup of Very Large Filesystems and Files 192 Simultaneous Backup of Many Clients to One Drive 192 Simultaneous Backup of Many Clients to One Drive 192 Simultaneous Backup of One Client to Many Drives 196 Page ix Data Requiring Special Treatment 202 Storage Management Features 205 Reduction in Network Traffic 208 Support of a Standard or Custom Backup Format 216 Ease of Administration 219 Security 222 Ease of Recovery 223 Protection of the Backup Index 225 Robustness 227 Automation 227 Volume Verification 228 Cost 229 Vendor 230 Conclusions 231 6. High Availability 232 What Is High Availability? 232 HA Building Blocks 238 Commercial HA Solutions 243 The Impact of an HA Solution 245 IV. Bare-Metal Backup & Recovery M ethods 247 IV. Bare-Metal Backup & Recovery M ethods 247 7. SunOS/Solaris 249 What About Fire? 250 Homegrown Bare-Metal Recovery 251 Recovering a SunOS/Solaris System 256 8. Linux 270 How It Works 270 A Sample Bare-Metal Recovery 275 9. Compaq True-64 Unix 282 Compaq's btcreate Utility 283 Homegrown Bare-Metal Recovery 284 10. HP-UX 290 HP's make_recovery Utility 291 The copyutil Utility 295 Using dump and restore 299 Page x 11. IRIX 306 SGI's Backup and Restore Utilities 307 System Recovery with Backup Tape 310 Homegrown Bare-Metal Recovery 315 12. AIX 323 IBM's mksysb Utility 324 IBM's Sysback/6000 Utility 330 IBM's Sysback/6000 Utility 330 System Cloning 337 V. Database Backup & Recovery 339 13. Backing Up Databases 341 Can It Be Done? 342 Confusion: The Mysteries of Database Architecture 343 The Muck Stops Here: Databases in Plain English 344 What's the Big Deal? 345 Database Structure 346 An Overview of a Page Change 360 What Can Happen to an RDBMS? 361 Backing Up an RDBMS 363 Restoring an RDBMS 370 Documentation and Testing 374 Unique Database Requirements 375 14. Informix Backup & Recovery 376 Informix Architecture 377 Automating Informix Startup: The dbstart.informix.sh Script 387 Protect the Physical Log, Logical Log, and sysmaster 392 Which Backup Utility Should I Use? 400 Physical Backups Without a Storage Manager: ontape 403 Physical Backups with a Storage Manager: onbar 424 Recovering Informix 428 Logical Backups 451 Logical Backups 451 15. Oracle Backup & Recovery 455 Oracle Architecture 455 Physical Backups Without a Storage Manager 463 Physical Backups with a Storage Manager 476 Managing the Archived Redologs 480 Page xi Recovering Oracle 483 Logical Backups 526 A Broken Record 529 16. Sybase Backup & Recovery 531 Sybase Architecture 531 Physical Backups Without a Storage Manager 542 Physical Backups with a Storage Manager 554 Recovering Sybase 554 Logical Backups 583 An Ounce of Prevention 586 VI. Backup & Recovery Potpourri 589 17. ClearCase Backup & Recovery 591 ClearCase Architecture 592 VOB Backup and Recovery Procedures 598 View Backup and Recovery Procedures 608 Summary 615 Summary 615 18. Backup Hardware 616 Choosing on a Backup Drive 616 Using Backup Hardware 621 Tape Drives 625 Optical Drives 635 Automated Backup Hardware 641 Vendors 643 Hardware Comparison 645 19. Miscellanea 649 Volatile Filesystems 649 Demystifying dump 654 Gigabit Ethernet 663 Disk Recovery Companies 664 Yesterday 664 Trust Me About the Backups 665 Index 667 Page xiii PREFACE Like many people, I had to learn backups the hard way. I worked at a large company where I was responsible for backing up Unix SVr3/4, Ultrix, HP-UX 8-10, AIX 3, Solaris 2.3, Informix, Oracle, and Sybase. In those days I barely understood how Unix worked, and I really didn't understand how databases worked-yet it was my responsibility to back it all up. I did what any normal person would do. I went to the biggest bookstore I could find and looked for a book on the subject. There weren't any books on the shelf, so I went to the counter where they could search the Books in Print database. Searching on the word "backup" brought up one book on how to back up Macintoshes. Disillusioned, I did what many other people did: I read the backup chapters in several system and database administration books. Even the best books covered it on only a cursory level, and none of them told me how to automate the backups of 200 Unix machines that ran eight different flavors of Unix and three different database products. Another common problem with these chapters is that they would dedicate 90 percent or more to backup and less than 10 percent to recovery. So my company did what many others had done before us-we reinvented the wheel and wrote our own homegrown utilities and procedures. Then one day I realized that our backup/recovery needs had outgrown our homegrown utilities, which meant that we needed to look at purchasing a commercial utility. Again, there were no resources to help explain the differences between the various backup utilities that were available at that time, so we did what most people do-we talked to the vendors. Since most of the vendors just bashed one another, our job was to try to figure out who was telling the truth and who wasn't. We then wrote a Request For Information (RFI) and a Request For Proposal (RFP) and sent it to the vendors we were considering, whose quotes ranged from Page xiv $16,000 to $150,000. Believe it or not, the least expensive product also did the best on the RFI, and we bought and installed our first commercial backup utility. The day came for me to leave my first backup utility behind, as I was hired by a company that would one day become Collective Technologies. Finally, a chance to get out of backups and become a real system administrator! Interestingly enough, one of my first clients had been performing backups only sporadically, but I discovered that they had a valid license for the commercial product with which I was already familiar. (Imagine the luck.) While rolling out that product, they asked me also to look at how they were backing up their Oracle databases. The next thing I knew, I had ported my favorite Oracle backup script and published it. The response to that article was amazing. People around the world wrote me and thanked me for sharing it, and I caught the publishing bug. One of Collective Technologies' mottos is, ''If something is broken, fix it!" Normally, we're talking about problems within our own company, but I applied it to the backup and recovery industry and the dream of this book was born. I Wish I Had This Book My dream was to write a book that would make sure that no one ever had to start from scratch again, and I believe that my coauthors and I have done just that. It contains every backup tool that I wish I had had when I first entered the Unix business and every lesson and trick that I've learned along the way. It covers how to back up and recover everything from a basic Unix workstation to a complicated Informix, Oracle, or Sybase database. Whether your budget barely stretches to cover the cost of the backup media or allows you to buy a silo bigger than your house, this book has something for you. Whether your task is to figure out how to back up, with no commercial utilities, an environment such as the one I first encountered or to choose from among more than 50 commercial backup utilities, this book will tell you how to do it. With that in mind, let me mention a few things about this book that are unique. [...]... for treating me as one of your own and supporting us on our quest Pahingi ng sinagong? To all the teachers who kept trying to get me to live up to my potential: You finally got through To Collective Technologies: I never could have done this if it hadn't been for you folks You truly are a special group of people, and I'm proud to be known as one of you To Ed Taylor, Gordon Galligher, Curt Vincent, and. .. travels, you welcomed me as one of your own Only you truly understand what it's like trying to do something like this, and I couldn't have done it without you To O' Reilly & Associates: Thank you for the opportunity to bring this much-needed book to market (Sorry it took me two and a half years longer than it should have!) To Gigi Estabrook, my editor: We'll have to actually meet one of these days! I don't... enter it in some sort of order processing program? What about things like budgets, memoranda, inventories, and any other "paperwork" that you throw around from day to day? Do you keep copies of every important memo that you send, or do you depend on the computer for that? If you're like most people, you have grown quite dependent on these things we call computers You forget how much of your work has been... Gustavo Vegas of Collective Technologies HP-UX bare-metal recovery Steve Ferguson of Collective Technologies IRIX bare-metal recovery Blayne Puklich of Collective Technologies Sybase backup and recovery Bryn Smith of Collective Technologies Without these folks, either the book would never have been completed or it would contain substantially less data than the book you see today Another group of people... lilo Chapter 9, Compaq True-64 Unix, covers both Compaq True-64 Unix' s bare-metal recovery tool and the Compaq version of the homegrown procedure covered in Chapter 7 Chapter 10 , HP-UX, covers the make _recovery tool, which now comes with HP-UX to perform bare-metal recoveries, along with the HP version of the homegrown procedure Chapter 11 , IRIX, explains how the different versions of IRIX's Backup and. .. themselves computer people Get input from other departments to answer this question When all those 1s and 0s come together, just what kind of stuff are we talking about? Do you use manual accounting methods, or are your company's financial records stored in some accounting software somewhere? When a customer calls in and orders something, do you jot that down on a carbon-copied order form, or do you enter... for future editions, by writing to: Page xix O' Reilly & Associates 10 1 Morris Street Sebastopol, CA 95472 1- 800-998-9938 (in the U.S or Canada) 1- 707-829-0 515 (international/local) 1- 707-829- 010 4 (fax) You can also send messages electronically To be put on our mailing list or to request a catalog, send email to: nuts@oreilly.com To ask technical questions or comment on the book, send email to: bookquestions@oreilly.com... bare-metal recovery Charles Gagnon and Brian Jensen of Collective Technologies AMANDA John R Jackson and Alexandre Oliva from the AMANDA Core Development Team Clearcase backup and recovery Bob Fulwiler of Seattle, Washington Compaq/Digital Unix bare-metal recovery Matthew Huff of Collective Technologies Page xx Dump internals David Young of Collective Technologies High-availability systems Josh Newcomb and. .. tangible and most devastating of all losses If you've got your entire customer database on a computer somewhere, how will you know who they are if that computer dies? So you might actually "lose" your customers and never find them again You also could lose customers who depend on data that is on one or more of your computers; if the customer finds out that you have lost his data, he will undoubtedly... limitations that could not be fixed without abandoning them altogether Those companies chose to develop custom, or even proprietary, backup methods They attempted to overcome the limitations that products that were based on dump and tar could not Not all of these proprietary backup products did well, however, which sometimes left customers in the lurch with scores of backup volumes that could be read only . & Recovery Utilities 18 5 5. Commercial Backup Utilities 18 7 What to Look For 18 8 Full Support of Your Platforms 18 9 Backup of Raw Partitions 19 1 Backup of Very Large Filesystems and Files 19 2 Simultaneous. O& apos;Leary Printing History: November 19 99: First Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O& apos ;Reilly logo are registered trademarks of O& apos ;Reilly & Associates, Inc. Many of the. to: nuts@oreilly.com To ask technical questions or comment on the book, send email to: bookquestions@oreilly.com This Book Was a Team Effort I have never worked with a group of people like the ones I work