Programming Python, 2nd Edition, O’Reilly IT-SC book 1 Programming Python, 2nd Edition Mark Lutz Publisher: O'Reilly 2nd Edition March 2001 ISBN: 0-596-00085-5, 1256 pages Programming Python focuses on advanced applications of Python. Endorsed by Python creator Guido van Rossum, it demonstrates advanced Python techniques, and addresses software design issues such as reusability and object- oriented programming. The enclosed platform-neutral CD- ROM (view CD-ROM content online at http://examples.oreilly.com/python2 ) has book examples and various Python-related packages, including the full Python Version 2.0 source code distribution. Programming Python, 2nd Edition, O’Reilly IT-SC book 2 Enjoy the life together. Programming Python, 2nd Edition, O’Reilly IT-SC book 3 Foreword Preface "And Now for Something Completely Different . . . Again" Signs of the Python Times Why This Edition? Major Changes in This Edition Using the Examples and Demos Conventions Used in This Book Where to Look for Updates Contacting O'Reilly Acknowledgments 1. Introducing Python 1.1 "And Now for Something Completely Different" 1.2 The Life of Python 1.3 The Compulsory Features List 1.4 What's Python Good For? 1.5 What's Python Not Good For? I: System Interfaces 2. System Tools 2.1 "The os.path to Knowledge" 2.2 Why Python Here? 2.3 System Scripting Overview 2.4 The sys Module 2.5 The os Module 2.6 Script Execution Context 2.7 Current Working Directory 2.8 Command-Line Arguments 2.9 Shell Environment Variables 2.10 Standard Streams 2.11 File Tools 2.12 Directory Tools 3. Parallel System Tools 3.1 "Telling the Monkeys What to Do" 3.2 Forking Processes 3.3 Threads 3.4 Program Exits 3.5 Interprocess Communication 3.6 Pipes 3.7 Signals 3.8 Launching Programs on Windows 3.9 Other System Tools 4. Larger System Examples I 4.1 "Splits and Joins and Alien Invasions" 4.2 Splitting and Joining Files 4.3 Generating Forward-Link Web Pages 4.4 A Regression Test Script 4.5 Packing and Unpacking Files Programming Python, 2nd Edition, O’Reilly IT-SC book 4 4.6 User-Friendly Program Launchers 5. Larger System Examples II 5.1 "The Greps of Wrath" 5.2 Fixing DOS Line Ends 5.3 Fixing DOS Filenames 5.4 Searching Directory Trees 5.5 Visitor: Walking Trees Generically 5.6 Copying Directory Trees 5.7 Deleting Directory Trees 5.8 Comparing Directory Trees II: GUI Programming 6. Graphical User Interfaces 6.1 "Here's Looking at You, Kid" 6.2 Python GUI Development Options 6.3 Tkinter Overview 6.4 Climbing the GUI Learning Curve 6.5 The End of the Tutorial 6.6 Python/Tkinter for Tcl/Tk Converts 7. A Tkinter Tour, Part 1 7.1 "Widgets and Gadgets and GUIs, Oh My!" 7.2 Configuring Widget Appearance 7.3 Toplevel Windows 7.4 Dialogs 7.5 Binding Events 7.6 Message and Entry 7.7 Checkbutton, Radiobutton, and Scale 7.8 Running GUI Code Three Ways 7.9 Images 8. A Tkinter Tour, Part 2 8.1 "On Today's Menu: Spam, Spam, and Spam" 8.2 Menus 8.3 Listboxes and Scrollbars 8.4 Text 8.5 Canvas 8.6 Grids 8.7 Time Tools, Threads, and Animation 8.8 The End of the Tour 8.9 The PyDemos and PyGadgets Launchers 9. Larger GUI Examples 9.1 "Building a Better Mouse Trap" 9.2 Advanced GUI Coding Techniques 9.3 Complete Program Examples 9.4 PyEdit: A Text Editor Program/Object 9.5 PyView: An Image and Notes Slideshow 9.6 PyDraw: Painting and Moving Graphics 9.7 PyClock: An Analog/Digital Clock Widget 9.8 PyToe: A Tic-Tac-Toe Game Widget Programming Python, 2nd Edition, O’Reilly IT-SC book 5 9.9 Where to Go from Here III: Internet Scripting 10. Network Scripting 10.1 "Tune in, Log on, and Drop out" 10.2 Plumbing the Internet 10.3 Socket Programming 10.4 Handling Multiple Clients 10.5 A Simple Python File Server 11. Client-Side Scripting 11.1 "Socket to Me!" 11.2 Transferring Files over the Net 11.3 Processing Internet Email 11.4 The PyMailGui Email Client 11.5 Other Client-Side Tools 12. Server-Side Scripting 12.1 "Oh What a Tangled Web We Weave" 12.2 What's a Server-Side CGI Script? 12.3 Climbing the CGI Learning Curve 12.4 The Hello World Selector 12.5 Coding for Maintainability 12.6 More on HTML and URL Escapes 12.7 Sending Files to Clients and Servers 13. Larger Web Site Examples I 13.1 "Things to Do When Visiting Chicago" 13.2 The PyMailCgi Web Site 13.3 The Root Page 13.4 Sending Mail by SMTP 13.5 Reading POP Email 13.6 Utility Modules 13.7 CGI Script Trade-offs 14. Larger Web Site Examples II 14.1 "Typos Happen" 14.2 The PyErrata Web Site 14.3 The Root Page 14.4 Browsing PyErrata Reports 14.5 Submitting PyErrata Reports 14.6 PyErrata Database Interfaces 14.7 Administrative Tools 14.8 Designing for Reuse and Growth 15. Advanced Internet Topics 15.1 "Surfing on the Shoulders of Giants" 15.2 Zope: A Web Publishing Framework 15.3 HTMLgen: Web Pages from Objects 15.4 JPython ( Jython): Python for Java 15.5 Grail: A Python-Based Web Browser 15.6 Python Restricted Execution Mode Programming Python, 2nd Edition, O’Reilly IT-SC book 6 15.7 XML Processing Tools 15.8 Windows Web Scripting Extensions 15.9 Python Server Pages 15.10 Rolling Your Own Servers in Python IV: Assorted Topics 16. Databases and Persistence 16.1 "Give Me an Order of Persistence, but Hold the Pickles" 16.2 Persistence Options in Python 16.3 DBM Files 16.4 Pickled Objects 16.5 Shelve Files 16.6 SQL Database Interfaces 16.7 PyForm: A Persistent Object Viewer 17. Data Structures 17.1 "Roses Are Red, Violets Are Blue; Lists Are Mutable, and So Is Class Foo" 17.2 Implementing Stacks 17.3 Implementing Sets 17.4 Binary Search Trees 17.5 Graph Searching 17.6 Reversing Sequences 17.7 Permuting Sequences 17.8 Sorting Sequences 17.9 Data Structures Versus Python Built-ins 17.10 PyTree: A Generic Tree Object Viewer 18. Text and Language 18.1 "See Jack Hack. Hack, Jack, Hack" 18.2 Strategies for Parsing Text in Python 18.3 String Module Utilities 18.4 Regular Expression Matching 18.5 Parser Generators 18.6 Hand-Coded Parsers 18.7 PyCalc: A Calculator Program/Object V: Integration 19. Extending Python 19.1 "I Am Lost at C" 19.2 C Extensions Overview 19.3 A Simple C Extension Module 19.4 The SWIG Integration Code Generator 19.5 Wrapping C Environment Calls 19.6 A C Extension Module String Stack 19.7 A C Extension Type String Stack 19.8 Wrapping C++ Classes with SWIG 20. Embedding Python 20.1 "Add Python. Mix Well. Repeat." 20.2 C Embedding API Overview 20.3 Basic Embedding Techniques Programming Python, 2nd Edition, O’Reilly IT-SC book 7 20.4 Registering Callback Handler Objects 20.5 Using Python Classes in C 20.6 ppembed: A High-Level Embedding API 20.7 Other Integration Topics VI: The End 21. Conclusion: Python and the Development Cycle 21.1 "That's the End of the Book, Now Here's the Meaning of Life" 21.2 "Something's Wrong with the Way We Program Computers" 21.3 The "Gilligan Factor" 21.4 Doing the Right Thing 21.5 Enter Python 21.6 But What About That Bottleneck? 21.7 On Sinking the Titanic 21.8 So What's Python: The Sequel 21.9 In the Final Analysis 21.10 Postscript to the Second Edition A. Recent Python Changes A.1 Major Changes in 2.0 A.2 Major Changes in 1.6 A.3 Major Changes Between 1.3 and 1.5.2 B. Pragmatics B.1 Installing Python B.2 Book Examples Distribution B.3 Environment Configuration B.4 Running Python Programs B.5 Python Internet Resources C. Python Versus C++ Colophon Programming Python, 2nd Edition, O’Reilly IT-SC book 8 Foreword Less than five years ago, I wrote the Foreword for the first edition of Programming Python. Since then, the book has changed about as much as the language and the Python community! I no longer feel the need to defend Python: the statistics and developments listed in Mark's Preface speak for themselves. In the past year, Python has made great strides. We released Python 2.0, a big step forward, with new standard library features such as Unicode and XML support, and several new syntactic constructs, including augmented assignment: you can now write x += 1 instead of x = x+1. A few people wondered what the big deal was (answer: instead of x, imagine dict[key] or list[index]), but overall this was a big hit with those users who were already used to augmented assignment in other languages. Less warm was the welcome for the extended print statement, print>>file, a shortcut for printing to a different file object than standard output. Personally, it's the Python 2.0 feature I use most frequently, but most people who opened their mouths about it found it an abomination. The discussion thread on the newsgroup berating this simple language extension was one of the longest ever-apart from the never-ending Python versus Perl thread. Which brings me to the next topic. (No, not Python versus Perl. There are better places to pick a fight than a Foreword.) I mean the speed of Python's evolution, a topic dear to the heart of the author of this book. Every time I add a feature to Python, another patch of Mark's hair turns gray-there goes another chapter out of date! Especially the slew of new features added to Python 2.0, which appeared just as he was working on this second edition, made him worry: what if Python 2.1 added as many new things? The book would be out of date as soon as it was published! Relax, Mark. Python will continue to evolve, but I promise that I won't remove things that are in active use! For example, there was a lot of worry about the string module. Now that string objects have methods, the string module is mostly redundant. I wish I could declare it obsolete (or deprecated) to encourage Python programmers to start using string methods instead. But given that a large majority of existing Python code-even many standard library modules-imports the string module, this change is obviously not going to happen overnight. The first likely opportunity to remove the string module will be when we introduce Python 3000; and even at that point, there will probably be a string module in the backwards compatibility library for use with old code. Python 3000?! Yes, that's the nickname for the next generation of the Python interpreter. The name may be considered a pun on Windows 2000, or a reference to Mystery Science Theater 3000, a suitably Pythonesque TV show with a cult following. When will Python 3000 be released? Not for a loooooong time-although you won't quite have to wait until the year 3000. Originally, Python 3000 was intended to be a complete rewrite and redesign of the language. It would allow me to make incompatible changes in order to fix problems with the language design that weren't solvable in a backwards compatible way. The current plan, however, is that the necessary changes will be introduced gradually Programming Python, 2nd Edition, O’Reilly IT-SC book 9 into the current Python 2.x line of development, with a clear transition path that includes a period of backwards compatibility support. Take, for example, integer division. In line with C, Python currently defines x/y with two integer arguments to have an integer result. In other words, 1/2 yields 0! While most dyed-in-the-wool programmers expect this, it's a continuing source of confusion for newbies, who make up an ever-larger fraction of the (exponentially growing) Python user population. From a numerical perspective, it really makes more sense for the / operator to yield the same value regardless of the type of the operands: after all, that's what all other numeric operators do. But we can't simply change Python so that 1/2 yields 0.5, because (like removing the string module) it would break too much existing code. What to do? The solution, too complex to describe here in detail, will have to span several Python releases, and involves gradually increasing pressure on Python programmers (first through documentation, then through deprecation warnings, and eventually through errors) to change their code. By the way, a framework for issuing warnings will be introduced as part of Python 2.1. Sorry, Mark! So don't expect the announcement of the release of Python 3000 any time soon. Instead, one day you may find that you are already using Python 3000-only it won't be called that, but rather something like Python 2.8.7. And most of what you've learned in this book will still apply! Still, in the meantime, references to Python 3000 will abound; just know that this is intentionally vaporware in the purest sense of the word. Rather than worry about Python 3000, continue to use and learn more about the Python version that you do have. I'd like to say a few words about Python's current development model. Until early 2000, there were hundreds of contributors to Python, but essentially all contributions had to go through my inbox. To propose a change to Python, you would mail me a context diff, which I would apply to my work version of Python, and if I liked it, I would check it into my CVS source tree. (CVS is a source code version management system, and the subject of several books.) Bug reports followed the same path, except I also ended up having to come up with the patch. Clearly, with the increasing number of contributions, my inbox became a bottleneck. What to do? Fortunately, Python wasn't the only open source project with this problem, and a few smart people at VA Linux came up with a solution: SourceForge! This is a dynamic web site with a complete set of distributed project management tools available: a public CVS repository, mailing lists (using Mailman, a very popular Python application!), discussion forums, bug and patch managers, and a download area, all made available to any open source project for the asking. We currently have a development group of 30 volunteers with SourceForge checkin privileges, and a development mailing list comprising twice as many folks. The privileged volunteers have all sworn their allegiance to the BDFL (Benevolent Dictator For Life-that's me :-). Introduction of major new features is regulated via a lightweight system of proposals and feedback called Python Enhancement Proposals (PEPs). Our PEP system proved so successful that it was copied almost verbatim by the Tcl community when they made a similar transition from Cathedral to Bazaar. Programming Python, 2nd Edition, O’Reilly IT-SC book 10 So, it is with confidence in Python's future that I give the floor to Mark Lutz. Excellent job, Mark. And to finish with my favorite Monty Python quote: Take it away, Eric, the orchestra leader! Guido van Rossum Reston, Virginia, January 2001 "And Now for Something Completely Different . . . Again" The first edition of this book was one of the first to present the Python language. This second edition is an almost completely new advanced Python topics book, designed to be a follow-up to the core language material in Learning Python and supplemented by the reference material in Python Pocket Reference. That is, this edition is focused on ways to use Python, rather than on the language itself. Python development concepts are explored along the way in fact, they really become meaningful only in the context of larger examples like those in this edition. But in general, this text now assumes that you already have at least a passing acquaintance with Python language fundamentals, and moves on to present the rest of the Python story. In this preface, I'll explain some of the rationales for this major rewrite, describe the structure of this edition in more detail, and give a brief overview of how to use the Python programs shipped on the enclosed CD-ROM (view CD-ROM content online at http://examples.oreilly.com/python2 ). First of all, though, a history lesson is in order. Signs of the Python Times It's been an exciting five years in the Python world. Since I wrote the first edition of this book between 1995 and 1996, Python has grown from a new kid on the scripting languages block to an established and widely used tool in companies around the world. Although measuring the popularity of an open source (http://opensource.org ) and freely distributed tool such as Python is not always easy, most statistics available reveal exponential growth in Python's popularity over the last five years. Among the most recent signs of Python's explosive growth: Books As I write this in 2001, there are now over a dozen Python books on the market, with almost that many more on the way (in 1995 there were none). Some of these books are focused on a particular domain (e.g., Windows), and some are available in German, French, and Japanese language editions. Users In 1999, one leading industry observer suggested that there were as many as 300,000 Python users worldwide, based on various statistics. Other estimates are more optimistic still. In early 2000, for instance, the Python web site was already on track to service 500,000 new Python interpreter downloads by year end (in addition [...]... widget PyToe An AI-powered graphical tic-tac-toe program PyForm A persistent object table browser PyCalc A calculator widget in Python/ Tk PyMail A Python/ Tk POP and SMTP email client PyFtp A simple Python/ Tk file-transfer GUI PyErrata A web-based error report system PyMailCgi A web-based email interface IT-SC book 18 Programming Python, 2nd Edition, O’Reilly There are also new mixed-mode C integration... independent implementation of the Python language that generates DLL and EXE files, allows Python code to be developed under Visual Studio, and provides seamless NET integration for Python scripts It promises to be a third IT-SC book 12 Programming Python, 2nd Edition, O’Reilly implementation of Python, along with the standard C-based Python, and the JPython Java-based system Education Python has also begun attracting... can get IT-SC book 31 Programming Python, 2nd Edition, O’Reilly Chapter 1 Introducing Python 1.1 "And Now for Something Completely Different" 1.2 The Life of Python 1.3 The Compulsory Features List 1.4 What's Python Good For? 1.5 What's Python Not Good For? 1.1 "And Now for Something Completely Different" This book is about using Python, a very high-level, object-oriented, open source[1] programming. .. as Python Major Changes in This Edition The best way to get a feel for any book is to read it, of course But especially for people who are familiar with the first edition, the next few sections go into more detail about what is new in this edition IT-SC book 14 Programming Python, 2nd Edition, O’Reilly It's Been Updated for Python 2.0 This edition has been updated for Python 2.0, and the Graphical User... Washington D.C., Colorado, Italy, IT-SC book 11 Programming Python, 2nd Edition, O’Reilly Korea, and England Such groups work on Python- related enhancements, organize Python events, and more Domains Python has grown to embrace both Microsoft Windows developers, with new support for COM and Active Scripting, as well as Java developers, with the new JPython (renamed "Jython") Java-based implementation of the... on IT-SC book 13 Programming Python, 2nd Edition, O’Reilly Why This Edition? One consequence of the growing popularity of Python has been an influx of new users, programming styles, and applications, all of which have conspired to make parts of the first edition of this book prime for updates Python itself has changed in smaller ways, but important extensions have simplified various aspects of Python. .. Python efforts underway as well For instance, ActiveState and PythonWare develop Python tools, O'Reilly (the publisher of this book) and a company named Foretech both organize annual Python conferences, and O'Reilly manages a supplemental Python web site (see the O'Reilly Network's Python DevCenter at http://www.oreillynet.com /python) The O'Reilly Python Conference is held as part of the annual Open Source... The Python language really does look like it was designed, not accumulated Python is also well tooled for modern software reuse methodologies In fact, writing highquality Python components that may be applied in multiple contexts is almost automatic Productivity IT-SC book 32 Programming Python, 2nd Edition, O’Reilly Python is optimized for speed of development It's easy to write programs fast in Python, ... also find Linux automated build scripts for the Python/ C integration examples in the top-level examples directory, which serve as indexes to major C examples Figure P-1 The PyDemos launcher with pop-ups and demos (Guido's photo reprinted with permission from Dr Dobb's Journal) IT-SC book 22 Programming Python, 2nd Edition, O’Reilly I've also included a top-level program called PyGadgets.py, and its relative... define it; Table 1-1 lists some of the common reasons cited for Python' s appeal Table 1-1 Python Language Features Features Benefits No compile or link steps Rapid development cycle turnaround No type declarations Simpler, shorter, and more flexible programs IT-SC book 35 Programming Python, 2nd Edition, O’Reilly Automatic memory management Garbage collection avoids bookkeeping code High-level datatypes . Programming Python, 2nd Edition, O’Reilly IT-SC book 1 Programming Python, 2nd Edition Mark Lutz Publisher: O'Reilly 2nd Edition March 2001 ISBN: 0-5 9 6-0 008 5-5 , 1256. full Python Version 2.0 source code distribution. Programming Python, 2nd Edition, O’Reilly IT-SC book 2 Enjoy the life together. Programming Python, 2nd Edition, O’Reilly IT-SC book. Objects 15.4 JPython ( Jython): Python for Java 15.5 Grail: A Python- Based Web Browser 15.6 Python Restricted Execution Mode Programming Python, 2nd Edition, O’Reilly IT-SC book 6 15.7