Programming Amazon Web Services by James Murty Publisher: O'Reilly Pub Date: March 15, 2008 Print ISBN-13: 978-0-59-651581-2 Pages: 600 Table of Contents | Index Overview Building on the success of its storefront and fulfillment services, Amazon now allows businesses to "rent" computing power, data storage and bandwidth on its vast network platform This book demonstrates how developers working with small- to mid-sized companies can take advantage of Amazon Web Services (AWS) such as the Simple Storage Service (S3), Elastic Compute Cloud (EC2), Simple Queue Service (SQS), Flexible Payments Service (FPS), and SimpleDB to build web-scale business applications With AWS, Amazon offers a new paradigm for IT infrastructure: use what you need, as you need it, and pay as you go Programming Web Services explains how you can access Amazon's open APIs to store and run applications, rather than spend precious time and resources building your own With this book, you'll learn all the technical details you need to: Store and retrieve any amount of data using application servers, unlimited data storage, and bandwidth with the Amazon S3 service Buy computing time using Amazon EC2's interface to requisition machines, load them with an application environment, manage access permissions, and run your image using as many or few systems as needed Use Amazon's web-scale messaging infrastructure to store messages as they travel between computers with Amazon SQS Leverage the Amazon FPS service to structure payment instructions and allow the movement of money between any two entities, humans or computers Create and store multiple data sets, query your data easily, and return the results using Amazon SimpleDB Scale up or down at a moment's notice, using these services to employ as much time and space as you need Whether you're starting a new online business, need to ramp up existing services, or require an offsite backup for your home, Programming Web Services gives you the background and the practical knowledge you need to start using AWS Other books explain how to build web services This book teaches businesses how to take make use of existing services from an established technology leader Programming Amazon Web Services by James Murty Publisher: O'Reilly Pub Date: March 15, 2008 Print ISBN-13: 978-0-59-651581-2 Pages: 600 Table of Contents | Index Copyright Dedication Preface Chapter 1 Infrastructure in the Cloud Section 1.1 Amazon Web Services for Infrastructure Section 1.2 Thinking Like Amazon Section 1.3 Reality Check Section 1.4 Interfaces: REST and Query Versus SOAP Chapter 2 Interacting with Amazon Web Services Section 2.1 REST-Based APIs Section 2.2 User Authentication Section 2.3 Performing AWS Requests Chapter 3 S3: Simple Storage Service Section 3.1 S3 Overview Section 3.2 Interacting with S3 Section 3.3 Buckets Section 3.4 Objects Section 3.5 Alternative Hostnames Section 3.6 Access Control Lists Section 3.7 Server Access Logging (Beta) Section 3.8 Signed URIs Section 3.9 Distributing Objects with BitTorrent Chapter 4 S3 Applications Section 4.1 Share Large Files Section 4.2 Online Backup with AWS::S3 Section 4.3 S3 Filesystem with ElasticDrive Section 4.4 Mediated Access to S3 with JetS3t Chapter 5 EC2: Elastic Compute Cloud (Beta) Section 5.1 EC2 Overview Section 5.2 Interacting with EC2 Section 5.3 Keypairs Section 5.4 Network Security by IP Section 5.5 Finding Amazon Machine Images Section 5.6 Controlling Instances Section 5.7 Log In to an Instance Section 5.8 Security Groups Section 5.9 Managing and Sharing AMIs Section 5.10 Console Output and Instance Reboot Chapter 6 Using EC2 Instances and Images Section 6.1 EC2 Instances in Detail Section 6.2 Data Management in EC2 Section 6.3 Modifying an AMI Section 6.4 Registering an AMI Section 6.5 Create an AMI from Scratch Chapter 7 EC2 Applications Section 7.1 Dynamic DNS Section 7.2 On-Demand VPN Server with OpenVPN Section 7.3 Web Photo Album with Gallery 2 Chapter 8 SQS: Simple Queue Service Section 8.1 SQS Overview Section 8.2 Interacting with SQS Section 8.3 Queues Section 8.4 Messages Section 8.5 Queue Attributes Section 8.6 Queue Access Control Chapter 9 SQS Applications Section 9.1 Messaging Simulator Section 9.2 Distributed Application Services with BOTO Section 9.3 Automated Management of EC2 Instance Pools with Lifeguard Chapter 10 FPS: Flexible Payments Service (Beta) Section 10.1 FPS Overview Section 10.2 Interacting with FPS Section 10.3 Managing Your Tokens Section 10.4 Acquiring Third-Party Tokens Section 10.5 Pay Now Widgets Chapter 11 FPS Transactions and Accounts Section 11.1 Performing FPS Transactions Section 11.2 Account Management and Information Chapter 12 FPS Advanced Topics Section 12.1 Gatekeeper Language Guide Section 12.2 Micropayments with FPS Section 12.3 Building a Marketplace Application Section 12.4 Subscribing to FPS Event Notifications Chapter 13 SimpleDB (Beta) Section 13.1 SimpleDB Overview Section 13.2 Interacting with SimpleDB Section 13.3 Domains Section 13.4 Items and Attributes Section 13.5 Representing Data in SimpleDB Section 13.6 Performing Queries Section 13.7 Stock Price Database: A Mini SimpleDB Application Appendix A AWS Resources Section A.1 AWS Online Resources Section A.2 Client Tools Section A.3 API Libraries Section A.4 Third-Party AWS Solutions Appendix B AWS API Error Codes Section B.1 S3: Simple Storage Service Section B.2 EC2: Elastic Compute Cloud Section B.3 SQS: Simple Queue Service Section B.4 FPS: Flexible Payments Service Section B.5 SimpleDB Colophon Index Copyright Copyright © 2008, James Murty All rights reserved Printed in the United States of America Published by O'Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O'Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://safari.oreilly.com) For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com Editor: Mike Loukides Production Editor: Sumita Mukherji Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly Media, Inc The image of an Atlantic wreckfish and related trade dress are trademarks of O'Reilly Media, Inc Many of the designations uses by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in this book, and O'Reilly Media, Inc was aware of a trademark claim, the designations have been printed in caps or initial caps While every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein For Bec Preface Amazon Web Services (AWS) is the name of a suite of web services made available by Amazon that allow third-party developers to access and build on the company's technology platform This suite includes a number of infrastructure services that can augment or replace the traditional physical infrastructure required by web applications These infrastructure services provide storage, computing power, a messaging system, a payment system, and a database that can be accessed by anyone with an Amazon.com account and a credit card Best of all, with these services you pay only for what you use Whether you are merely experimenting with the services or using them as a platform for web applications serving thousands of users, you rent only the infrastructure you need and only when you need it These services offer a compelling alternative to building your applications on top of standard, physical hardware, because they can provide the scalable, reliable, and cost-effective infrastructure components you need without the expense or effort involved in managing your own hardware They give you the freedom to concentrate on your application instead of the physical platform it runs on They also significantly reduce the initial investment required to build and offer a service, while giving you confidence that your application will be able to not only survive any sudden attention it receives, but to grow and thrive in response By leaving the infrastructure management to Amazon, you can focus on developing your application and growing your business With the expense of infrastructure removed from the equation, anyone building web applications can compete and succeed based on their innovation, execution, and ideas instead of being limited by their resources Jeff Bezos, the CEO of Amazon, has famously referred to IT infrastructure and its attendant setup, management, and expense headaches as "muck." In declaring "We make muck so you don't have to," Bezos has offered to handle the mucky, infrastructure side of web applications and allow the rest of us to focus on the interesting side Perhaps you should take him up on the offer P.1 What's in This Book? In this book we explore the set of web services offered by Amazon that provide pay-as-you-go, virtual infrastructure, and we look at the practicalities of designing and building applications to take advantage of Amazon's infrastructure services We will provide detailed sample code that shows how to interact with the services' application program interfaces (APIs), and we will discuss how the services can be used individually and in combination to build systems in the real world The first two chapters introduce the services we will use in this book and discuss how they work at the HTTP messaging level: Chapter 1 In this chapter, we introduce Amazon's infrastructure services and discuss the main advantages and disadvantages of using online services to meet your infrastructure needs To use Amazon's services effectively, you may have to adjust your mindset and expectations: things work differently in the cloud Chapter 2 To interact with the Amazon Web Services, you will need a communications library to create request messages that are structured to be readable to the services, and authenticated so that only you can access your protected data You will also need to understand the response messages the services return In this chapter, we describe how to construct and interpret the HTTP messages used by Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] paid AMIs partitions (EC2) Pay Now Widgets 2nd Pay operation payment instructions [See tokens] payment methods PaymentInstruction element payments [See FPS transactions] PeekMessage operation permissions policy documents constructing generating special characters POST requests defined Query API in SQS vs GET requests 2nd vs PUT requests web browsers postpaid payment instruments balance retrieval creating description listing risks settling a debt writing off a debt predicates prepaid payment instruments balance retrieval creating description funding liability retrieval listing PrepaidBalance element PrepaidInstrumentIds element pretty-print (pp) library private variables product codes 2nd 3rd 4th propagation latency 2nd 3rd 4th PUT requests defined vs POST requests REST API 2nd 3rd web browsers PutAttributes operation Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] queries, SimpleDB case sensitivity language performing syntax vs SQL queries, XPath Query API components parameters request description strings REST vs sending requests signing requests SOAP vs versioning Query operation queue URL queues access control attributes in boto creating deleting identifying listing overview Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] RDBMS vs SimpleDB reaction time interval RebootInstances operation ReceiveMessage operation recipient tokens 2nd recipients Recurring tokens 2nd 3rd redirects 2nd 3rd Refund operation refunding transactions 2nd 3rd RegisterImage API RegisterImage operation RemoveGrant operation request body request description strings 2nd 3rd request headers 2nd request signatures 2nd 3rd RequestId element Reserve operation resources API libraries client tools websites response body response headers ResponseBatchSize element REST API authenticating requests components Query vs sending requests signing requests SOAP vs XML documents and RestImageAttribute operation restoring files retrieval, conditional RetryTransaction operation 2nd RevokeSecurityGroupIngress operation REXML 2nd Ruby package installation server certificate checking 2nd Unicode character support Ruby libraries for FPS for HTTP loading for S3 for XML documents 2nd 3rd Ruby objects, mapping XML to RunInstances operation Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] S3 (Simple Storage Service) access control lists alternative hostnames 2nd API libraries APIs 2nd 3rd architecture AWS-based services for AWS-specific headers AWS::S3 library 2nd benefits binary data and BitTorrent and client tools cost 2nd data storage data verification downloading files ElasticDrive error messages file backup 2nd 3rd file sharing forms home page Host headers initializing connection limitations 2nd overview 2nd policy documents 2nd query parameters restoring files Server Access Logging signed URIs storage sharing subdomains uploading files URI construction 2nd user identification sandbox 2nd 3rd 4th Secret Access Key 2nd secure HTTPS flag Secure Shell connections 2nd 3rd security groups allowing access via creating deleting listing for OpenVPN removing access via rules security risks 2nd 3rd 4th 5th securityGroupInfo element sender tokens 2nd 3rd senders SendMessage operation Server Access Logging configuring 2nd log files modifying settings overview retrieving settings server certificates 2nd server load 2nd Service Level Agreements (SLAs) service, generic bootstrapping boto vs class description features message format visibility reset SetQueueAttributes operation Settle operation SettleDebt operation 2nd SetupPostpaid SetupPrepaid SHA-1 (Secure Hash Algorithm) signed URIs Simple Queue Service (SQS) [See SQS (Simple Queue Service)] Simple Storage Service (S3) [See S3 (Simple Storage Service)] SimpleDB APIs 2nd automated encoding caching data encoding data representation error messages home page non-ASCII data in overview 2nd pricing 2nd [See also box usage] queries vs RDBMS response messages sample application size, estimating structure SingleUse tokens SingleUse tokens SingleUse tokens 2nd SLAs (Service Level Agreements) slash (/) sleep time SOAP API authenticating requests components Query vs REST vs SQL vs SimpleDB SQS (Simple Queue Service) advantages API libraries APIs 2nd architecture attributes client types development guidelines error messages grantees home page Lifeguard and limitations overview pricing stacks starts-with (condition) StartTimeForNextTransaction element startup scripts AMI creation bootstrap 2nd boto Cockpit Lite status change history 2nd 3rd Status element storage sharing subdomains 2nd 3rd SubscribeForCallerNotification operation subscription payment model synchronous responses system clock synchronization 2nd 3rd system services and EC2 instances Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] technical support, for AWS TerminateInstances operation third party accounts thread pools time adjustment 2nd 3rd time offset time values, encoding Token element TokenFriendlyName element tokens [See also specific token types] cancellation cancellation by third party defined defining expiration installing 2nd limiting 2nd 3rd 4th listing modifying retrieving third-party [See CBUI pipeline] types of usage history viewing TokenUsageLimits element Transaction element transaction fees TransactionParts element TransactionResponse element transactions [See FPS transactions] Transactions element truncated listings 2nd 3rd Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] Unable to create bucket message Unicode character support Universal Resource Identifiers (URIs) components expiration in S3 2nd signed Unrestricted tokens 2nd 3rd UnSubscribeForCallerNotification operation uploading files 2nd 3rd user authentication 2nd Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] virtual block devices virtual hosting Virtual Private Networks visibility 2nd visibility timeout 2nd 3rd 4th Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] WAR files web archive (WAR) files Web Service Definition Language WEBrick servlet library widgets WriteOffDebt operation Index [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] x-amz- (prefix) x-amz-meta- (prefix) X.509 certificates XML documents in Amazon Web Services creating as error messages mapping to Ruby objects REXML Ruby libraries for XML elements XPath ... usually includes the title, author, publisher, and ISBN For example: "Programming Amazon Web Services: S3, EC2, SQS, FPS, and SimpleDB by James Murty Copyright 2008 James Murty, 978-0-596-51581-2." If you feel your use of code examples falls outside fair use or... http://www.aws .amazon. com /s3 Elastic Compute Cloud (EC2) http://www.aws .amazon. com /ec2 Simple Queue Service (SQS) http://www.aws .amazon. com /sqs Flexible Payments Service (FPS) http://www.aws .amazon. com /fps SimpleDB http://www.aws .amazon. com/sdb... researching the technical details of the Amazon Web Services and understanding how best to use them in the real world To the extent that this book provides an accurate and useful reference for developers or users of Amazon Web Services, I