Table of Contents Preface Who this book is for Versions Organization . Conventions used in this book . Differences between the first edition and second edition Comments and questions Hal's acknowledgments from the first edition Acknowledgments for the second edition . 1 2 2 3 4 5 5 6 6 1. Networking Fundamentals . 1.1 Networking overview 1.2 Physical and data link layers 1.3 Network layer 1.4 Transport layer 1.5 The session and presentation layers . 9 9 11 12 18 19 2. Introduction to Directory Services . 2.1 Purpose of directory services 2.2 Brief survey of common directory services 2.3 Name service switch 2.4 Which directory service to use . 24 24 25 29 29 3. Network Information Service Operation 3.1 Masters, slaves, and clients . 3.2 Basics of NIS management . 3.3 Files managed under NIS 3.4 Trace of a key match 31 31 34 41 52 4. System Management Using NIS . 4.1 NIS network design 4.2 Managing map files 4.3 Advanced NIS server administration 4.4 Managing multiple domains 56 56 58 65 67 5. Living with Multiple Directory Servers . 5.1 Domain name servers . 5.2 Implementation . 5.3 Fully qualified and unqualified hostnames 5.4 Centralized versus distributed management . 5.5 Migrating from NIS to DNS for host naming 5.6 What next? . 70 70 72 74 76 77 77 6. System Administration Using the Network File System . 6.1 Setting up NFS 6.2 Exporting filesystems . 6.3 Mounting filesystems . 6.4 Symbolic links 6.5 Replication 6.6 Naming schemes . 78 79 80 85 96 99 103 7. Network File System Design and Operation 7.1 Virtual filesystems and virtual nodes . 7.2 NFS protocol and implementation . 7.3 NFS components . 7.4 Caching 7.5 File locking . 7.6 NFS futures . 108 108 109 117 122 127 129 8. Diskless Clients 8.1 NFS support for diskless clients . 8.2 Setting up a diskless client 8.3 Diskless client boot process 8.4 Managing client swap space . 8.5 Changing a client's name 8.6 Troubleshooting 8.7 Configuration options 8.8 Brief introduction to JumpStart administration 8.9 Client/server ratios 132 132 133 136 140 142 143 147 150 151 9. The Automounter 9.1 Automounter maps 9.2 Invocation and the master map . 9.3 Integration with NIS . 9.4 Key and variable substitutions 9.5 Advanced map tricks 9.6 Side effects . 153 154 162 167 169 173 182 10. PC/NFS Clients . 10.1 PC/NFS today 10.2 Limitations of PC/NFS 10.3 Configuring PC/NFS 10.4 Common PC/NFS usage issues 10.5 Printer services 184 184 185 188 189 191 11. File Locking . 11.1 What is file locking? 11.2 NFS and file locking 11.3 Troubleshooting locking problems 192 192 194 196 12. Network Security 12.1 User-oriented network security 12.2 How secure are NIS and NFS? . 12.3 Password and NIS security . 12.4 NFS security . 12.5 Stronger security for NFS 12.6 Viruses . 200 200 206 207 210 223 245 13. Network Diagnostic and Administrative Tools 13.1 Broadcast addresses 13.2 MAC and IP layer tools 13.3 Remote procedure call tools . 13.4 NIS tools 13.5 Network analyzers . 247 248 250 268 276 283 14. NFS Diagnostic Tools 14.1 NFS administration tools . 14.2 NFS statistics . 14.3 snoop . 14.4 Publicly available diagnostics . 14.5 Version 2 and Version 3 differences . 14.6 NFS server logging . 14.7 Time synchronization . 295 295 298 307 311 317 318 331 15. Debugging Network Problems 15.1 Duplicate ARP replies . 15.2 Renegade NIS server . 15.3 Boot parameter confusion 15.4 Incorrect directory content caching . 15.5 Incorrect mount point permissions 15.6 Asynchronous NFS error messages . 335 335 337 338 339 343 345 16. Server-Side Performance Tuning . 16.1 Characterization of NFS behavior 16.2 Measuring performance 16.3 Benchmarking 16.4 Identifying NFS performance bottlenecks . 16.5 Server tuning . 349 349 351 352 353 357 17. Network Performance Analysis . 17.1 Network congestion and network interfaces . 17.2 Network partitioning hardware 17.3 Network infrastructure 17.4 Impact of partitioning . 17.5 Protocol filtering 367 367 369 371 372 374 18. Client-Side Performance Tuning 18.1 Slow server compensation 18.2 Soft mount issues 18.3 Adjusting for network reliability problems 18.4 NFS over wide-area networks . 18.5 NFS async thread tuning . 18.6 Attribute caching 18.7 Mount point constructions 18.8 Stale filehandles . 376 376 381 382 384 385 387 388 390 A. IP Packet Routing . A.1 Routers and their routing tables . A.2 Static routing 392 392 396 B. NFS Problem Diagnosis . B.1 NFS server problems B.2 NFS client problems . B.3 NFS errno values . 397 397 398 399 C. Tunable Parameters 401 Colophon 405 Managing NFS and NIS 1 Preface Twenty years ago, most computer centers had a few large computers shared by several hundred users. The "computing environment" was usually a room containing dozens of terminals. All users worked in the same place, with one set of disks, one user account information file, and one view of all resources. Today, local area networks have made terminal rooms much less common. Now, a "computing environment" almost always refers to distributed computing, where users have personal desktop machines, and shared resources are provided by special-purpose systems such as file, computer, and print servers. Each desktop requires redundant configuration files, including user information, network host addresses, and local and shared remote filesystem information. A mechanism to provide consistent access to all files and configuration information ensures that all users have access to the "right" machines, and that once they have logged in they will see a set of files that is both familiar and complete. This consistency must be provided in a way that is transparent to the users; that is, a user should not know that a filesystem is located on a remote fileserver. The transparent view of resources must be consistent across all machines and also consistent with the way things work in a non-networked environment. In a networked computing environment, it's usually up to the system administrator to manage the machines on the network (including centralized servers) as well as the network itself. Managing the network means ensuring that the network is transparent to users rather than an impediment to their work. The Network File System (NFS) and the Network Information Service (NIS) [1] provide mechanisms for solving "consistent and transparent" access problems. The NFS and NIS protocols were developed by Sun Microsystems and are now licensed to hundreds of vendors and universities, not to mention dozens of implementations from the published NFS and NFS specifications. NIS centralizes commonly replicated configuration files, such as the password file, on a single host. It eliminates duplicate copies of user and system information and allows the system administrator to make changes from one place. NFS makes remote filesystems appear to be local, as if they were on disks attached to the local host. With NFS, all machines can share a single set of files, eliminating duplicate copies of files on different machines in the network. Using NFS and NIS together greatly simplifies the management of various combinations of machines, users, and filesystems. [1] NIS was formerly called the "Yellow Pages." While many commands and directory names retain the yp prefix, the formal name of the set of services has been changed to avoid conflicting with registered trademarks. NFS provides network and filesystem transparency because it hides the actual, physical location of the filesystem. A user's files could be on a local disk, on a shared disk on a fileserver, or even on a machine located across a wide-area network. As a user, you're most content when you see the same files on all machines. Just having the files available, though, doesn't mean that you can access them if your user information isn't correct. Missing or inconsistent user and group information will break Unix file permission checking. This is where NIS complements NFS, by adding consistency to the information used to build and describe the shared filesystems. A user can sit down in front of any workstation in his or her group that is running NIS and be reasonably assured that he or she can log in, find his or her home directory, and access tools such as compilers, window systems, and publishing packages. In addition to making life easier for the users, NFS and NIS simplify the tasks of Managing NFS and NIS 2 system administrators, by centralizing the management of both configuration information and disk resources. NFS can be used to create very complex filesystems, taking components from many different servers on the network. It is possible to overwhelm users by providing "everything everywhere," so simplicity should rule network design. Just as a database programmer constructs views of a database to present only the relevant fields to an application, the user community should see a logical collection of files, user account information, and system services from each viewpoint in the computing environment. Simplicity often satisfies the largest number of users, and it makes the system administrator's job easier. Who this book is for This book is of interest to system administrators and network managers who are installing or planning new NFS and NIS networks, or debugging and tuning existing networks and servers. It is also aimed at the network user who is interested in the mechanics that hold the network together. We'll assume that you are familiar with the basics of Unix system administration and TCP/IP networking. Terms that are commonly misused or particular to a discussion will be defined as needed. Where appropriate, an explanation of a low-level phenomenon, such as Ethernet congestion will be provided if it is important to a more general discussion such as NFS performance on a congested network. Models for these phenomena will be drawn from everyday examples rather than their more rigorous mathematical and statistical roots. This book focuses on the way NFS and NIS work, and how to use them to solve common problems in a distributed computing environment. Because Sun Microsystems developed and continues to innovate NFS and NIS, this book uses Sun's Solaris operating system as the frame of reference. Thus if you are administering NFS on non-Solaris systems, you should use this book in conjunction with your vendor's documentation, since utilities and their options will vary by implementation and release. This book explains what the configuration files and utilities do, and how their options affect performance and system administration issues. By walking through the steps comprising a complex operation or by detailing each step in the debugging process, we hope to shed light on techniques for effective management of distributed computing environments. There are very few absolute constraints or thresholds that are universally applicable, so we refrain from stating them. This book should help you to determine the fair utilization and performance constraints for your network. Versions This book is based on the Solaris 8 implementations of NFS and NIS. When used without a version number, "Solaris" refers to the Solaris 2.x, Solaris 7, and Solaris 8 operating systems and their derivatives (note that the next version of Solaris after Solaris 2.6 was Solaris 7; in the middle of the development process, Sun renamed Solaris 2.7 to Solaris 7). NFS- and NIS- related tools have changed significantly between Solaris 2.0 and Solaris 8, so while it is usually the case that an earlier version of Solaris supports a function we discuss, it is not infrequent that it will not. For example, early releases of Solaris 2.x did not even have true NIS support. For another, Sun has made profound enhancements to NFS with nearly every release of Solaris. Managing NFS and NIS 3 The Linux examples presented throughout the book were run on the Linux 2.2.14-5 kernel. Linux kernels currently implement NFS Version 2, although a patch is available that provides Version 3 support. Organization This book is divided into two sections. The first twelve chapters contain explanations of the implementation and operation of NFS and NIS. Chapter 13 through Chapter 18 cover advanced administrative and debugging techniques, performance analysis, and tuning. Building on the introductory material, the second section of the book delves into low-level details such as the effects of network partitioning hardware and the various steps in a remote procedure call. The material in this section is directly applicable to the ongoing maintenance and debugging of a network. Here's the chapter-by-chapter breakdown: • Chapter 1 provides an introduction to the underlying network protocols and services used by NFS and NIS. • Chapter 2 provides a survey of the popular directory services. • Chapter 3 discusses the architecture of NIS and its operation on both NIS servers and NIS clients. The focus is on how to set up NIS and its implementation features that affect network planning and initial configuration. • Chapter 4 discusses operational aspects of NIS that are important to network administrators. This chapter explores common NIS administration techniques, including map management, setting up multiple NIS domains, and using NIS with domain name services. • Chapter 5 explains the issues around using both NIS and the Directory Name Service (DNS) on the same network. • Chapter 6 covers basic NFS operations, such as mounting and exporting filesystems. • Chapter 7 explains the architecture of NFS and the underlying virtual filesystem. It also discusses the implementation details that affect performance, such as file attributes and data caching. • Chapter 8 is all about diskless clients. It also presents debugging techniques for clients that fail to boot successfully. • Chapter 9 discusses the automounter, a powerful but sometimes confusing tool that integrates NIS administrative techniques and NFS filesystem management. • Chapter 10 covers PC/NFS, a client-side implementation of NFS for Microsoft Windows machines. • Chapter 11 focuses on file locking and how it relates to NFS. • Chapter 12 explores network security. Issues such as restricting access to hosts and filesystems form the basis for this chapter. We'll also go into how to make NFS more secure, including a discussion of setting up NFS security that leverages encryption for stronger protection. • Chapter 13 describes the administrative and diagnostic tools that are applied to the network and its systems as a whole. This chapter concentrates on the network and on interactions between hosts on the network, instead of the per-machine issues presented in earlier chapters. Tools and techniques are described for analyzing each layer in the protocol stack, from the Ethernet to the NFS and NIS applications. • Chapter 14 focuses on tools used to diagnose NFS problems. • Chapter 15 describes how to debug common network problems. Managing NFS and NIS 4 • Chapter 16 discusses how to tune your NFS and, to a lesser extent, NIS servers. • Chapter 17 covers performance tuning and analysis of machines and the network. • Chapter 18 explores NFS client tuning, including NFS mount parameter adjustments. • Appendix A explains how IP packets are forwarded to other networks. It is additional background information for discussions of performance and network configuration. • Appendix B summarizes NFS problem diagnosis using the NFS statistics utility and the error messages printed by clients experiencing NFS failures. • Appendix C summarizes parameters for tuning NFS performance and other attributes. Conventions used in this book Font and format conventions for Unix commands, utilities, and system calls are: • Excerpts from script or configuration files will be shown in a constant-width font: 192.9.200.1 bitatron • Sample interactive sessions, showing command-line input and corresponding output, will be shown in a constant-width font, with user-supplied input in bold: • % ls foobar • If the command can be typed by any user, the percent sign ( % ) will be shown as the prompt. If the command must be executed by the superuser, then the pound sign ( # ) will be shown as the prompt: # /usr/sbin/ypinint -m • If a particular command must be typed on a particular machine, the prompt will include a hostname: bitatron# mount wahoo:/export /mnt • Inside of an excerpt from a script, configuration file, or other ASCII file, the pound sign will be used to indicate the beginning of a comment (unless the configuration file requires a different comment character, such as an asterisk (*)): • # • #Hal's machine 192.9.200.1 bitatron • Unix commands and command lines are printed in italics when they appear in the body of a paragraph. For example, the ls command lists files in a directory. • Hostnames are printed in italics. For example, server wahoo contains home directories. • Filenames are printed in italics, for example, the /etc/passwd file. • NIS map names and mount options are printed in italics. The passwd map is used with the /etc/passwd file, and the timeo mount option changes NFS client behavior. • System and library calls are printed in italics, with parentheses to indicate that they are C routines. For example, the gethostent( ) library call locates a hostname in an NIS map. • Control characters will be shown with a CTRL prefix, for example, CTRL-Z. Managing NFS and NIS 5 Differences between the first edition and second edition The first edition was based on SunOS 4.1, whereas this edition is based on Solaris 8. The second edition covers much more material, mostly due to the enhancements made to NFS, including a new version of NFS (Version 3), a new transport protocol for NFS (TCP/IP), new security options (IPsec and Kerberos V5), and also more tools to analyze your systems and network. The second edition also drops or sharply reduces the following material from the first edition (all chapter numbers and titles are from the first edition): • Chapter 4. Systems and networks are now bigger, faster, and more complicated. We believe the target reader will be more interested in administering NIS and NFS, rather than writing applications based on NIS. • Chapter 9. At the time the second edition was written, most people were accessing their electronic mail boxes using the POP or IMAP protocols. A chapter focused on using NFS to access mail would appeal but to a small minority. • Chapter 14. This chapter survives in the second edition, but it is much smaller. This is because there are more competing PC/NFS products available than before, and also because many people who want to share files between PCs and Unix servers run the open source Samba package on their Unix servers. Still, there are some edge conditions that justify PC/NFS, so we discuss those, as well as general PC/NFS issues. • Appendix A. When this appendix was written, local area networks were much less reliable than they are today. The shift to better and standard technology, even low technology like Category 5 connector cables, has made a big difference. Thus, given the focus on software administration, there's not much practical use for presenting such material in this edition. • Appendix D. The NFS Benchmark appendix in the first edition explained how to use the nhfsstone benchmark, and was relevant in the period of NFS history when there was no standard, industry-recognized benchmark. Since the first edition, the Standard Performance Evaluation Corporation (SPEC) has addressed the void with its SFS benchmark (sometimes referred to as LADDIS). The SFS benchmark provides a way for prospective buyers of an NFS server to compare it to others. Unfortunately, it's not practical for the target reader to build the complex test beds necessary to get good SFS benchmark numbers. A better alternative is to take advantage of the fact that SPEC lets anyone browse reported SFS results from its web site (http://www.spec.org/). Comments and questions We have tested and verified all the information in this book to the best of our abilities, but you may find that features have changed or that we have let errors slip through the production of the book. Please let us know of any errors that you find, as well as suggestions for future editions, by writing to: O'Reilly & Associates, Inc. 101 Morris St. Sebastopol, CA 95472 (800) 998-9938 (in the U.S. or Canada) (707) 829-0515 (international/local) (707) 829-0104 (fax) Managing NFS and NIS 6 You can also send messages electronically. To be put on our mailing list or to request a catalog, send email to: info@oreilly.com To ask technical questions or to comment on the book, send email to: bookquestions@oreilly.com We have a web site for the book, where we'll list examples, errata, and any plans for future editions. You can access this page at: http://www.oreilly.com/catalog/nfs2/ For more information about this book and others, see the O'Reilly web site: http://www.oreilly.com/ Hal's acknowledgments from the first edition This book would not have been completed without the help of many people. I'd like to thank Brent Callaghan, Chuck Kollars, Neal Nuckolls, and Janice McLaughlin (all of Sun Microsystems); Kevin Sheehan (Kalli Consulting); Vicki Lewolt Schulman (Auspex Systems); and Dave Hitz (H&L Software) for their neverending stream of answers to questions about issues large and small. Bill Melohn (Sun) provided the foundation for the discussion of computer viruses. The discussion of NFS performance tuning and network configuration is based on work done with Peter Galvin and Rick Sabourin at Brown University. Several of the examples of NIS and NFS configuration were taken from a system administrator's guide to NFS and NIS written by Mike Loukides for Multiflow Computer Company. The finished manuscript was reviewed by: Chuck Kollars, Mike Marotta, Ed Milstein, and Brent Callaghan (Sun); Dave Hitz (H&L Software); Larry Rogers (Princeton University); Vicky Lewold Schulman (Auspex); Simson Garfinkel (NeXTWorld); and Mike Loukides and Tim O'Reilly (O'Reilly & Associates, Inc.). This book has benefited in many ways from their insights, comments, and corrections. The production group of O'Reilly & Associates also deserves my gratitude for applying the finishing touches to this book. I owe a tremendous thanks to Mike Loukides of O'Reilly & Associates who helped undo four years of liberal arts education and associated writing habits. It is much to Mike's credit that this book does not read like a treatise on Dostoevsky's Crime and Punishment. [2] [2] I think I will cause my freshman composition lecturer pain equal to the credit given to Mike, since she assured me that reading and writing about Crime and Punishment would prepare me for writing assignments the rest of my life. I have yet to see how, except possibly when I was exploring performance issues. Acknowledgments for the second edition Thanks to Pat Parseghian (Transmeta), Marc Staveley (Sun), and Mike Loukides (O'Reilly & Associates, Inc.) for their input to the outline of the second edition. [...]... HP's HP-UX, IBM, AIX, and Linux NIS+ is much more secure than NIS This rest of this book ignores NIS+ and LDAP, and focuses on NIS and to some degree DNS, since those are what you are most likely to encounter If you are concerned about security, you'll need to seriously consider deploying NIS+ until LDAP catches up If security is not a concern, then NIS is fine 30 Managing NFS and NIS Chapter 3 Network... domains, 26 Managing NFS and NIS whereas NIS shared the limitations of the early Internet's hosts.txt file: flat and not very dynamic We will go into much more detail on how NIS operates in Chapter 3 2.2.3 NIS+ In 1992, Sun Microsystems released NIS+ with Solaris 2.0 Despite its name, NIS+ was more different than it was similar to NIS NIS+ was developed to address several deficiencies in NIS: Hierarchical... no server Moreover, LDAP is more complex to administer than other directory services NIS is perhaps the easiest to administer, but it is also the most limited It is, however, the universal directory for Unix systems 29 Managing NFS and NIS DNS is the standard for hostnames and addresses, and you'll find it handy for accessing hosts outside your domain NIS+ has gained some acceptance among other non-Solaris... encouragement, understanding, and awesome support throughout the writing of this book Thank you for putting up with my late hours, work weekends, and late dinner dates 8 Managing NFS and NIS Chapter 1 Networking Fundamentals The Network Information Service (NIS) and Network File System (NFS) are services that allow you to build distributed computing systems that are both consistent in their appearance and transparent... hierarchical operation, and NIS+ 's attributes for security and simple update X.500 differs from DNS, NIS, and NIS+ in the following ways: 27 Managing NFS and NIS • X.500 is very explicit about what each level of the hierarchy of a domain name looks like If you see a domain name like: chicago.manufacturing.widget.com it could easily be a DNS, NIS, or NIS+ name, and it could very well be for a host or a domain... need to know how the underlying services work The lower-level network protocols are quite complex, and several books have been written about them without even touching on NFS and NIS services 9 Managing NFS and NIS Therefore, this chapter contains only a brief outline of the network services used by NFS and NIS Network protocols are typically described in terms of a layered model, in which the protocols... all of the desktop workstations and servers, then NFS and NIS will not function properly Even though NFS or NIS will appear "broken," the real issue is with a lower level in the network stack The following sections briefly describe the function of each layer and the mapping of NFS and NIS into them Many books have been written about the ISO seven-layer model, TCP/IP, and Ethernet, so their treatment... divorce NIS+ servers have the capability to accept incremental updates, which allows the updates to be more efficiently distributed 2.2.4 X.500 Around the same time DNS and NIS were being designed and deployed, the International Standards Organization (ISO) started meeting to define an ISO standard directory, called X.500 X.500 shares DNS's and NIS+ 's attributes for hierarchical operation, and NIS+ 's... three times and started a family It was pretty clear to me that the state of networking in general, and NFS and NIS in particular, was moving much faster than I was, and the only way this second edition became possible was to hand over the reins Mike Eisler and Ricardo Labiaga have done a superb job of bridging the technical eon since the first edition, and I thank them deeply for their patience and volumes... NIS, and then look at how to install NIS on each type of machine 3.1 Masters, slaves, and clients NIS is built on the client-server model An NIS server is a host that contains NIS data files, called maps Clients are hosts that request information from these maps Servers are further divided into master and slave servers: the master server is the true single owner of the map data Slave NIS servers handle . (NFS) and the Network Information Service (NIS) [1] provide mechanisms for solving "consistent and transparent" access problems. The NFS and NIS. weekends, and late dinner dates. Managing NFS and NIS 9 Chapter 1. Networking Fundamentals The Network Information Service (NIS) and Network File System (NFS)