Programming the Perl DBI Alligator Descartes & Tim Bunce First Edition February 2000 ISBN: 1-56592-699-4, 350 pages The primary interface for database programming in Perl is DBI Programming the Perl DBI is coauthored by Alligator Descartes, one of the most active members of the DBI community, and by Tim Bunce, the inventor of DBI The book explains the architecture of DBI, shows you how to write DBIbased programs and explains both DBI's nuances and the peculiarities of each individual DBD This is the definitive book for database programming in Perl Table of Contents Preface 1 Introduction From Mainframes to Workstations Perl DBI in the Real World A Historical Interlude and Standing Stones Basic Non-DBI Databases Storage Managers and Layers Query Languages and Data Functions Standing Stones and the Sample Database Flat-File Databases Putting Complex Data into Flat Files Concurrent Database Access and Locking DBM Files and the Berkeley Database Manager The MLDBM Module Summary SQL and Relational Databases The Relational Database Methodology Datatypes and NULL Values Querying Data Modifying Data Within Tables Creating and Destroying Tables 41 Programming with the DBI DBI Architecture Handles Data Source Names Connection and Disconnection Error Handling Utility Methods and Functions 57 Interacting with the Database Issuing Simple Queries Executing Non-SELECT Statements Binding Parameters to Statements Binding Output Columns do( ) Versus prepare( ) Atomic and Batch Fetching 76 Advanced DBI Handle Attributes and Metadata Handling LONG/LOB Data Transactions, Locking, and Isolation 97 Table of Contents (cont ) ODBC and the DBI ODBC-Embraced and Extended DBI-Thrashed and Mutated The Nuts and Bolts of ODBC ODBC from Perl The Marriage of DBI and ODBC Questions and Choices Moving Between Win32::ODBC and the DBI And What About ADO? 116 DBI Shell and Database Proxying dbish-The DBI Shell Database Proxying 122 A DBI Specification 131 B Driver and Database Characteristics 171 C ASLaN Sacred Site Charter 249 Colophon 250 Author Interview 251 Description One of the greatest strengths of the Perl programming language is its ability to manipulate large amounts of data Database programming is therefore a natural fit for Perl, not only for business applications but also for CGI-based web and intranet applications The primary interface for database programming in Perl is DBI DBI is a database-independent package that provides a consistent set of routines regardless of what database product you use Oracle, Sybase, Ingres, Informix, you name it The design of DBI is to separate the actual database drivers (DBDs) from the programmer's API, so any DBI program can work with any database, or even with multiple databases by different vendors simultaneously Programming the Perl DBI is coauthored by Alligator Descartes, one of the most active members of the DBI community, and by Tim Bunce, the inventor of DBI For the uninitiated, the book explains the architecture of DBI and shows you how to write DBI-based programs For the experienced DBI dabbler, this book reveals DBI's nuances and the peculiarities of each individual DBD The book includes: • An introduction to DBI and its design • How to construct queries and bind parameters • Working with database, driver, and statement handles • Debugging techniques • Coverage of each existing DBD • A complete reference to DBI This is the definitive book for database programming in Perl Programming the Perl DBI Preface The DBI is the standard database interface for the Perl programming language The DBI is databaseindependent, which means that it can work with just about any database, such as Oracle, Sybase, Informix, Access, MySQL, etc While we assume that readers of this book have some experience with Perl, we don't assume much familiarity with databases themselves The book starts out slowly, describing different types of databases and introducing the reader to common terminology This book is not solely about the DBI - it also concerns the more general subject of storing data in and retrieving data from databases of various forms As such, this book is split into two related, but standalone, parts The first part covers techniques for storing and retrieving data without the DBI, and the second, much larger part, covers the use of the DBI and related technologies Throughout the book, we assume that you have a basic grounding in programming with Perl and can put together simple scripts without instruction If you don't have this level of Perl awareness, we suggest that you read some of the Perl books listed in Section P.1 Once you're ready to read this book, there are some shortcuts that you can take depending on what you're most interested in reading about If you are interested solely in the DBI, you can skip Chapter without too much of a problem On the other hand, if you're a wizard with SQL, then you should probably skip Chapter to avoid the pain of us glossing over many fine details Chapter is a comparison between the DBI and ODBC and is mainly of interest to database geeks, design aficionados, and those people who have Win32::ODBC applications and are desperately trying to port them to DBI Here's a rundown of the book, chapter by chapter: Chapter This introduction sets up the general feel for the book Chapter This chapter covers the basics of storing and retrieving data either with core Perl functions through the use of delimited or fixed-width flat-file databases, or via non-DBI modules such as AnyDBM_File, Storable, Data::Dumper and friends Although the DBI isn't used in this chapter, the way the Storable and Data::Dumper modules are used to pack Perl data structures into strings can easily be applied to the DBI Chapter This chapter is a basic overview of SQL and relational databases and how you can write simple but powerful SQL statements to query and manipulate your database If you already know some SQL, you can skip this chapter If you don't know SQL, we advise you to read this chapter since the later chapters assume you have a basic knowledge of SQL and relational databases Chapter This chapter introduces the DBI to you by discussing the architecture of the DBI and basic DBI operations such as connecting to databases and handling errors This chapter is essential reading and describes the framework that the DBI provides to let you write simple, powerful, and robust programs Chapter This chapter is the meat of the DBI topic and discusses manipulating the data within your database - that is, retrieving data already stored in your database, inserting new data, and deleting and updating existing data We discuss the various ways in which you can perform these operations from the simple "get it working" stage to more advanced and optimized techniques for manipulating data page Programming the Perl DBI Chapter This chapter covers more advanced topics within the sphere of the DBI such as specifying attributes to fine-tune the operation of DBI within your applications, working with LONG/LOB datatypes, statement and database metadata, and finally transaction handling Chapter This chapter discusses the differences in design between DBI and ODBC, the other portable database API And, of course, this chapter highlights why DBI is easier to program with Chapter This chapter covers two topics that aren't exactly part of the core DBI, per se, but are extremely useful to know about First, we discuss the DBI shell, a command-line tool that allows you to connect to databases and issue arbitrary queries Second, we discuss the proxy architecture that the DBI can use, which, among other things, allows you to connect scripts on one machine to databases on another machine without needing to install any database networking software For example, you can connect a script running on a Unix box to a Microsoft Access database running on a Microsoft Windows box Appendix A This appendix contains the DBI specification, which is distributed with DBI.pm Appendix B This appendix contains useful extra information on each of the commonly used DBDs and their corresponding databases Appendix C This appendix contains the charter for the Ancient Sacred Landscape Network, which focuses on preserving sites such as the megalithic sites used for examples in this book Resources To help you navigate some of the topics in this book, here are some resources that you might want to check out before, during, and after reading this book: http://www.symbolstone.org/technology/perl/DBI The DBI home page This site contains lots of useful information about DBI and where to get the various modules from It also has links to the very active dbi-users mailing list and archives http://www.perl.com/CPAN This site includes the Comprehensive Perl Archive Network multiplexer, upon which you find a whole host of useful modules including the DBI An Introduction to Database Systems, by C J Date This book is the standard textbook on database systems and is highly recommended reading A Guide to the SQL Standard, by C J Date and Hugh Darwen An excellent book that's detailed but small and very readable http://w3.one.net/~jhoffman/sqltut.htm http://www.jcc.com/SQLPages/jccs_sql.htm http://www.contrib.andrew.cmu.edu/~shadow/sql.html These web sites contain information, specifications, and links on the SQL query language, of which we present a primer in Chapter Further information can be found by entering "SQL tutorial" or similar expressions into your favorite web search engine Learning Perl, by Randal Schwartz and Tom Christiansen A hands-on tutorial designed to get you writing useful Perl scripts as quickly as possible Exercises (with complete solutions) accompany each chapter A lengthy new chapter introduces you to CGI programming, while touching also on the use of library modules, references, and Perl's object-oriented constructs page Programming the Perl DBI Programming Perl, by Larry Wall, Tom Christiansen, and Randal Schwartz The authoritative guide to Perl version 5, the scripting utility that has established itself as the programming tool of choice for the World Wide Web, Unix system administration, and a vast range of other applications Version of Perl includes object-oriented programming facilities The book is coauthored by Larry Wall, the creator of Perl The Perl Cookbook, by Tom Christiansen and Nathan Torkington A comprehensive collection of problems, solutions, and practical examples for anyone programming in Perl Topics range from beginner questions to techniques that even the most experienced of Perl programmers will learn from More than just a collection of tips and tricks, The Perl Cookbook is the long-awaited companion volume to Programming Perl, filled with previously unpublished Perl arcana Writing Apache Modules with Perl and C, by Lincoln Stein and Doug MacEachern This book teaches you how to extend the capabilities of your Apache web server regardless of whether you use Perl or C as your programming language The book explains the design of Apache, mod_perl, and the Apache API From a DBI perspective, it discusses the Apache::DBI module, which provides advanced DBI functionality in relation to web services such as persistent connection pooling optimized for serving databases over the Web Boutell FAQ (http://www.boutell.com/faq/) and others These links are invaluable to you if you want to deploy DBI-driven web sites They explain the dos and don'ts of CGI programming in general MySQL & mSQL, by Randy Jay Yarger, George Reese, and Tim King For users of the MySQL and mSQL databases, this is a very useful book It covers not only the databases themselves but also the DBI drivers and other useful topics like CGI programming Typographical Conventions The following font conventions are used in this book: Constant Width is used for method names, function names, variables, and attributes It is also used for code examples Italic is used for filenames, URLs, hostnames, and emphasis Code Examples You are invited to copy the code in the book and adapt it for your own needs Rather than copying by hand, however, we encourage you to download the code from http://www.oreilly.com/catalog/perldbi/ page Programming the Perl DBI How to Contact Us We have tested and verified all the information in this book to the best of our abilities, but you may find that features have changed or that we have let errors slip through the production of the book Please let us know of any errors that you find, as well as suggestions for future editions, by writing to: O'Reilly & Associates, Inc 101 Morris St Sebastopol, CA 95472 1-800-998-9938 (in the U.S or Canada) 1-707-829-0515 (international/local) 1-707-829-0104 (fax) You can also send messages electronically To be put on our mailing list or to request a catalog, send email to: info@oreilly.com To ask technical questions or to comment on the book, send email to: bookquestions@oreilly.com We have a web site for the book, where we'll list examples, errata, and any plans for future editions You can access this page at: http://www.oreilly.com/catalog/perldbi/ For more information about this book and others, see the O'Reilly web site: http://www.oreilly.com Acknowledgments Alligator would like to thank his wife, Carolyn, for putting up with his authorial melodramatics and flouncing during the writing of this book Martin McCarthy should also get his name in lights for proofreading far too many of the early drafts of the book Phil Kizer also deserves a credit for running the servers that the DBI web site has sat on between 1995 and early 1999 Karin and John Attwood, Andy Burnham, Andy Norfolk, Chris Tweed, and many others on the stones mailing list deserve thanks (and beer) for aiding the preservation and presentation of many of the megalithic sites around the UK Further thanks to the people behind ASLaN for volunteering to a difficult job, and doing it well Tim would like to thank his wife, Máire, for being his wife; Larry Wall for giving the world Perl; Ted Lemon for having the idea that was, many years later, to become the DBI, and for running the mailing list for many of those years Thanks also to Tim O'Reilly for nagging me to write a DBI book, to Alligator for actually starting to it and then letting me jump on board (and putting up with my pedantic tendencies), and to Linda Mui for being a great editor The DBI has a long history[1] and countless people have contributed to the discussions and development over the years First, we'd like to thank the early pioneeers including Kevin Stock, Buzz Moschetti, Kurt Andersen, William Hails, Garth Kennedy, Michael Peppler, Neil Briscoe, David Hughes, Jeff Stander, and Forrest D Whitcher [1] It all started on September 29, 1992 Then, of course, there are the poor souls who have struggled through untold and undocumented obstacles to actually implement DBI drivers Among their ranks are Jochen Wiedmann, Jonathan Leffler, Jeff Urlwin, Michael Peppler, Henrik Tougaard, Edwin Pratomo, Davide Migliavacca, Jan Pazdziora, Peter Haworth, Edmund Mergl, Steve Williams, Thomas Lowery, and Phlip Plumlee Without them, the DBI would not be the practical reality it is today We would both like to thank the many reviewers to gave us valuable feedback Special thanks to Matthew Persico, Nathan Torkington, Jeff Rowe, Denis Goddard, Honza Pazdziora, Rich Miller, Niamh Kennedy, Randal Schwartz, and Jeffrey Baker page Programming the Perl DBI Chapter Introduction The subject of databases is a large and complex one, spanning many different concepts of structure, form, and expected use There are also a multitude of different ways to access and manipulate the data stored within these databases This book describes and explains an interface called the Perl Database Interface, or DBI, which provides a unified interface for accessing data stored within many of these diverse database systems The DBI allows you to write Perl code that accesses data without needing to worry about database- or platform-specific issues or proprietary interfaces We also take a look at non-DBI ways of storing, retrieving, and manipulating data with Perl, as there are occasions when the use of a database might be considered overkill but some form of structured data storage is required To begin, we shall discuss some of the more common uses of database systems in business today and the place that Perl and DBI takes within these frameworks 1.1 From Mainframes to Workstations In today's computing climate, databases are everywhere In previous years, they tended to be used almost exclusively in the realm of mainframe-processing environments Nowadays, with pizza-box sized machines more powerful than room-sized machines of ten years ago, high-performance database processing is available to anyone In addition to cheaper and more powerful computer hardware, smaller database packages have become available, such as Microsoft Access and mSQL These packages give all computer users the ability to use powerful database technology in their everyday lives The corporate workplace has also seen a dramatic decentralization in database resources, with radical downsizing operations in some companies leading to their centralized mainframe database systems being replaced with a mixture of smaller databases distributed across workstations and PCs The result is that developers and users are often responsible for the administration and maintenance of their own databases and datasets This trend towards mixing and matching database technology has some important downsides Having replaced a centralized database with a cluster of workstations and multiple database types, companies are now faced with hiring skilled administration staff or training their existing administration staff for new skills In addition, administrators now need to learn how to glue different databases together It is in this climate that a new order of software engineering has evolved, namely databaseindependent programming interfaces If you thought administration staff had problems with downsizing database technology, developers may have been hit even harder A centralized mainframe environment implies that database software is written in a standard language, perhaps COBOL or C, and runs only on one machine However, a distributed environment may support multiple databases on different operating systems and processors, with each development team choosing their preferred development environment (such as Visual Basic, PowerBuilder, Oracle Pro*C, Informix E/SQL, C++ code with ODBC - the list is almost endless) Therefore, the task of coordinating and porting software has rapidly gone from being relatively straightforward to extremely difficult Database-independent programming interfaces help these poor, beleagured developers by giving them a single, unified interface with which they can program This shields the developer from having to know which database type they are working with, and allows software written for one database type to be ported far more easily to another database For example, software originally written for a mainframe database will often run with little modification on Oracle databases Software written for Informix will generally work on Oracle with little modification And software written for Microsoft Access will usually run with little modification on Sybase databases page .. .Programming the Perl DBI Alligator Descartes & Tim Bunce First Edition February 2000 ISBN: 1-56592-699-4, 350 pages The primary interface for database programming in Perl is DBI Programming. .. reference to DBI This is the definitive book for database programming in Perl Programming the Perl DBI Preface The DBI is the standard database interface for the Perl programming language The DBI is... introduces you to CGI programming, while touching also on the use of library modules, references, and Perl' s object-oriented constructs page Programming the Perl DBI Programming Perl, by Larry Wall,