Software Engineering for Internet Applications Eve Andersson, Philip Greenspun, and Andrew Grumet After completing this self-contained course on server-based Internet applications software, students who start with only the knowledge of how to write and debug a computer program will have learned how to build Web-based applications on the scale of Amazon.com. Unlike the desktop applications that most students have already learned to build, server-based applications have multiple simultaneous users. This fact, coupled with the unreliability of networks, gives rise to the problems of concurrency and transactions, which students learn to manage by using the relational database system. After working their way to the end of the book, students will have the skills to take vague and ambitious specifications and turn them into a system design that can be built and launched in a few months. They will be able to test prototypes with end-users and refine the application design. They will understand how to meet the challenge of extreme business requirements with automatic code generation and the use of open- source toolkits where appropriate. Students will understand HTTP, HTML, SQL, mobile browsers, VoiceXML, data modeling, page flow and interaction design, server-side scripting, and usability analysis. The book, which originated as the text for an MIT course, is suitable for classroom use and will be a useful reference for software professionals developing multi-user Internet applications. It will also help managers evaluate such commercial software as Microsoft Sharepoint of Microsoft Content Management Server. Eve Andersson is Senior Vice President and Chair of the Bachelor of Science in Computer Science at Neumont University, Salt Lake City. Philip Greenspun, a software developer, author, teacher, pilot, and photographer, originated the Software Engineering for Internet Applications course at MIT. He is the author of Philip and Alex’s Guide to Web Publishing . Andrew Grumet received his Ph.D. in Electrical Engineering and Computer Science from MIT and builds Web applications as an independent software developer. Software Engineering for Internet Applications Eve Andersson Philip Greenspun Andrew Grumet Andersson, Greenspun, and Grumet Software Engin eering for Interne t Applications “Filled with practical advice for elegant and effective Web sites.” — Edward Tufte, author of The Visual Display of Quantitative Information computer science/software engineering 0-262-51191-6 The MIT Press Massachusetts Institute of Technology Cambridge, Massachusetts 02142 http://mitpress.mit.edu Cover design by Erin Hasley Software Engineering for Internet Applications Eve Andersson, Philip Greenspun, and Andrew Grumet Software Engineering for Internet Applications The MIT Press Cambridge, Massachusetts London, England 6 2006 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any elec- tronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, please email special_sales@mitpress.mit.edu or write to Special Sales Department, The MIT Press, 55 Hayward Street, Cambridge, MA 02142. This book was set in Times New Roman on 3B2 by Asco Typesetters, Hong Kong, and printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Andersson, Eve Astrid. Software engineering for Internet applications / Eve Andersson, Philip Greenspun, and Andrew Grumet. p. cm. Includes bibliographical references and index. ISBN 0-262-51191-6 (pbk. : alk. paper) 1. Internet programming. 2. Application software. 3. Software engineering. I. Greenspun, Philip. II. Grumet, Andrew. III. Title. QA76.625.A55 2006 005.2 0 76—dc22 2005049144 10987654321 Contents Preface vii Acknowledgments ix 1 Introduction 1 2 Basics 9 3 Planning 47 4 Software Structure 63 5 User Registration and Management 75 6 Content Management 97 7 Software Modularity 141 8 Discussion 161 9 Adding Mobile Users to Your Community 183 10 Voice (VoiceXML) 199 11 Scaling Gracefully 213 12 Search 241 13 Planning Redux 261 14 Distributed Computing with HTTP, XML, SOAP, and WSDL 269 15 Metadata (and Automatic Code Generation) 281 16 User Activity Analysis 303 17 Writeup 313 Reference Chapters A HTML 329 B Engagement Management by Cesar Brea 351 C Grading Standards 359 Glossary 363 To the Instructor 375 Sample Contract (between Student Team and Client) 391 About the Authors 393 Index 395 vi Contents Preface This is the textbook for the MIT course ‘‘Softw are Engineering for Internet Applications.’’ The course is intended for juniors and seniors in computer science. We assume that they know how to write a computer program and debug it. We do not assume knowledge of any particular programming lan- guages, standards, or protocols. The most concise statement of the course goal is that ‘‘The student finishes knowing how to build amazon.com by him or herself.’’ Other people who might find this book useful include the following: m professional software developers building online communities or other multi- user Internet applications m managers who are evaluating packaged software aimed at supporting online communities—various chapters contain criteria for judging the features of products such as Microsoft Sharepoint or Microsoft Content Management Server m university students and faculty looking to add some structure to a ‘‘capstone’’ project at the end of a computer science degree If you’re confuse d by the ‘‘student knows how to build amazon.com’’ state- ment, we can break it down in terms of principles and skills. The fundamental di¤erence between server-based Internet applications and the desktop appli- cations that students have already learned to build is that server-based appli- cations have multiple simultaneous users. Coupled with the unreliability of networks, this gives rise to the problems of concurrency and transactions. Stateless communications protocols such as HTTP mean that the student must learn how to build a stateful user experience on top of stateless protocols. For persistence between clicks and management of concurrency and transactions, the student needs to learn how to use the relational database management sys- tem. Finally, though this goes bey ond the simple stand-alone amazon.com-style service, students ought to learn about object-oriented distribute d computing where each object is a Web service. In addition to learning these principles, we’d like the student to learn some skills. This is a laboratory course, and we want students who graduate to be competent software engineers. We’d like our students to be able to take vague and ambitious specifications and turn them into a system design that can be built and launched within a few months, with the features most important to users and easiest to develop built first and the di‰cult bells and whistles de- ferred to a second version. We’d like our students to know how to test proto- types with end-users and refine their application design once or twice within even a three-month project. When business requirements are extreme, for example, ‘‘build me amazon.com by yourself in three months,’’ we want our students to understand how to cope with the challenge via automatic code gen- eration and use of open-source toolkits where appropriate. We can recast the ‘‘student knows how to build amazon.com’’ statement in terms of technologies used. By the time someone has finished reading and doing the exercises in this book, he or she will understand HTTP, HTML, SQL, mo- bile browsers on telephones, VoiceXML, data modeling, page flow and interac- tion design, server-side scripting, and usability analysis. Eve Andersson, Philip Greenspun, Andrew Grumet Cambridge, Massachusetts December 2005 viii Preface Acknowledgments The book is an outgrowth of six semesters of teaching experience at MIT and other universities. So our first thanks must go to our students, who taught us what worked and what didn’t work. It is a privilege to teach at MIT, and every instructor should have the opportunity once in a lifetime. We did not teach alone. Hal Abelson and the late Michael Dertouzos were our partners on the lecture podium. Hal was Mr. Pedagogy and also pushed the distributed computing ideas to the fore. Michael gave us an early push into voice applications. Lydia Sandon was our first teaching assistant. Ben Adida was our teaching assistant at MIT in the fall of 2003 when this book took its final pre-print shakedown cruise. In semesters where we did not have a full-time teaching assistant, the stu- dents’ most valuable partners were their industry mentors, most of whom were MIT alumni volunteering their time: David Abercrombie, Tracy Adams, Ben Adida, Mike Bonnet, Christian Brechbuhler, James Buszard-Welcher, Bryan Che, Bruce Keilin, Chris McEniry, Henry Minsky, Neil Mayle, Dan Parker, Richard Perng, Lydia Sandon, Mike Shurpik, Steve Strassman, Jessica Wong, and certainly a few more whose names have slipped from our memory. We’ve gotten valuable feedback from instructors at other universities using these materials, notably Aurelius Prochazka at Caltech and Oscar Bonilla at Universidad Galileo. [...]... California; voice is your only option There are some easy-to-identify missing features in typical Web-based applications For example, shareable and portable sessions You can use the Internet to share your photos You can use the Internet to share your music You can use the Internet to share your documents The one thing that you can’t typically share on the Internet is your experience of using the Internet. .. environment for the rest of the semester If you’ve been building RDBMS-backed Internet applications for some time, you can just use whatever you’ve been using Switching tools is seldom a path to glory If you haven’t built this kind of software before, read on Choosing an RDBMS There are probably three reasonable choices for an RDBMS this semester: (1) Microsoft SQL Server, (2) Oracle, and (3) PostgreSQL For. .. to learn Email and ‘‘reference library’’ were the two universally appealing applications of the Internet, according to a December 1999 survey conducted by Norman Nie and Lutz Erbring and reported in ‘ Internet and Society,’’ a January 2000 report of the Stanford Institute for the Quantitative Study of Society (http://www.stanford.edu/group/siqss/Press_Release/Preliminary_Report.pdf ) Entertainment and... good tool for storing tables of information? Consider first a spreadsheet program These are inexpensive and easy to use One should never apply more complex technology than necessary for solving a problem Something like Visicalc, Lotus 1-2-3, Microsoft Excel, or StarO‰ce Calc would seem to serve nicely The problem with a spreadsheet program is that it is designed for one user The program listens for user... browsers, via either the cookie extension to HTTP or URL rewriting Now you have to figure out a way to keep associated information on the Web server For flexibility in how you present and analyze user-contributed data, you’ll probably want to keep the information in a structured form For example, it would be nice to have a table of all the items put into shopping carts by various users And another table... 45 are responding to a discussion forum posting You don’t want the photo.net server to have forgotten your identity and demand your username and password again This presents you, the engineer, with a challenge: creating a stateful user experience on top of a fundamentally stateless protocol 12 Chapter 2 See the HTTP standard at http://www.w3.org/Protocols/ for more information on HTTP Where can you... The concern for man and his destiny must always be the chief interest of all technical e¤ort Never forget it between your diagrams and equations —Albert Einstein A twelve-year-old can build a nice Web application using the tools that came standard with any Linux or Windows machine Thus it is worth asking ourselves, ‘‘What is challenging, interesting, and inspiring about Internet- based applications? ’’... if you could go to a server-based information system and say ‘‘show me a listing of all the unanswered questions posted by other users’’? You might be willing to answer a few, simply for the satisfaction of helping another person and feeling like an expert When you 6 Chapter 1 got tired, you’d go to bed Teaching is fun if you don’t have to do it forty hours per week for thirty years Imagine if every... keyboard and mouse Once we’ve sent information into the closet, how do we get it back out? We can write down a request for a report on a strip of paper starting with ‘‘SELECT’’ and slide it under the door The DBMS in the dark closet will prepare a report for us and slide that back to us under the same door How do we evaluate whether or not a DBMS is powerful enough for our application? Starting in the... days for a report In 1979 Dan Bricklin (MIT EECS ’73) and Bob Frankston (MIT EECS ’70) developed Visicalc and suddenly most of the people who’d been hollering for programming services were able to build their own models With an RDBMS the metaphoric little strips of paper pushed under the door are declarative programs in the SQL language (See SQL for Web Nerds at http://philip.greenspun.com/sql/ for . Software Engineering for Internet Applications Eve Andersson, Philip Greenspun, and Andrew Grumet After completing this self-contained course on server-based Internet applications software, . his Ph.D. in Electrical Engineering and Computer Science from MIT and builds Web applications as an independent software developer. Software Engineering for Internet Applications Eve Andersson. design by Erin Hasley Software Engineering for Internet Applications Eve Andersson, Philip Greenspun, and Andrew Grumet Software Engineering for Internet Applications The MIT Press Cambridge, Massachusetts London,