Amazon Simple Storage Service Developer Guide

Table of Contents What Is Amazon S3? .......................................................................................................................... 1 How Do I...? ............................................................................................................................... 1 Introduction ...................................................................................................................................... 2 Overview of Amazon S3 and This Guide ....................................................................................... 2 Advantages to Amazon S3 .......................................................................................................... 2 Amazon S3 Concepts .................................................................................................................. 3 Buckets ............................................................................................................................. 3 Objects ............................................................................................................................. 3 Keys ................................................................................................................................. 4 Regions ............................................................................................................................. 4 Amazon S3 Data Consistency Model ..................................................................................... 4 Features .................................................................................................................................... 6 Reduced Redundancy Storage .............................................................................................. 7 Bucket Policies ................................................................................................................... 7 AWS Identity and Access Management .................................................................................. 8 Access Control Lists ............................................................................................................ 8 Versioning ......................................................................................................................... 8 Operations ........................................................................................................................ 8 Amazon S3 Application Programming Interfaces (API) ..................................................................... 8 The REST Interface ............................................................................................................. 9 The SOAP Interface ............................................................................................................ 9 Paying for Amazon S3 ................................................................................................................ 9 Related Services ......................................................................................................................... 9 Making Requests .............................................................................................................................. 11 About Access Keys .................................................................................................................... 11 AWS Account Access Keys .................................................................................................. 11 IAM User Access Keys ........................................................................................................ 12 Temporary Security Credentials .......................................................................................... 12 Request Endpoints .................................................................................................................... 13 Making Requests over IPv6 ........................................................................................................ 13 Getting Started with IPv6 .................................................................................................. 13 Using IPv6 Addresses in IAM Policies .................................................................................. 14 Testing IP Address Compatibility ........................................................................................ 15 Using DualStack Endpoints ............................................................................................... 15 Making Requests Using the AWS SDKs ........................................................................................ 19 Using AWS Account or IAM User Credentials ........................................................................ 19 Using IAM User Temporary Credentials ............................................................................... 26 Using Federated User Temporary Credentials ....................................................................... 36 Making Requests Using the REST API .......................................................................................... 49 DualStack Endpoints (REST API) ........................................................................................ 50 Virtual Hosting of Buckets ................................................................................................. 50 Request Redirection and the REST API ................................................................................ 54 Buckets ........................................................................................................................................... 57 Creating a Bucket ..................................................................................................................... 58 About Permissions ............................................................................................................ 59 Accessing a Bucket ................................................................................................................... 59 Bucket Configuration Options .................................................................................................... 60 Restrictions and Limitations ....................................................................................................... 61 Rules for Naming ............................................................................................................. 62 Examples of Creating a Bucket ................................................................................................... 63 Using the Amazon S3 Console ........................................................................................... 63 Using the AWS SDK for Java .............................................................................................. 63 Using the AWS SDK for .NET .............................................................................................. 64 Using the AWS SDK for Ruby Version 2 ............................................................................... 66 API Version 20060301 iii Amazon Simple Storage Service Developer Guide Using Other AWS SDKs ..................................................................................................... 66 Deleting or Emptying a Bucket .................................................................................................. 66 Delete a Bucket ................................................................................................................ 66 Empty a Bucket ................................................................................................................ 69 Bucket Website Configuration .................................................................................................... 70 Using the AWS Management Console ................................................................................. 71 Using the SDK for Java ..................................................................................................... 71 Using the AWS SDK for .NET .............................................................................................. 73 Using the SDK for PHP ..................................................................................................... 75 Using the REST API .......................................................................................................... 77 Transfer Acceleration ................................................................................................................ 77 Why use Transfer Acceleration? .......................................................................................... 77 Getting Started ................................................................................................................ 78 Requirements for Using Amazon S3 Transfer Acceleration ...................................................... 79 Transfer Acceleration Examples .......................................................................................... 80 Requester Pays Buckets ............................................................................................................. 89 Configure with the Console ............................................................................................... 89 Configure with the REST API ............................................................................................. 90 DevPay and Requester Pays ............................................................................................... 92 Charge Details ................................................................................................................. 92 Access Control ......................................................................................................................... 92 Billing and Usage Reporting ...................................................................................................... 93 Billing Reports ................................................................................................................. 93 Usage Report ................................................................................................................... 94 Understanding Billing and Usage Reports ............................................................................ 96 Using Cost Allocation Tags .............................................................................................. 100 Objects .......................................................................................................................................... 102 Object Key and Metadata ........................................................................................................ 103 Object Keys ................................................................................................................... 103 Object Metadata ............................................................................................................. 105 Storage Classes ...................................................................................................................... 107 Subresources .......................................................................................................................... 110 Versioning ............................................................................................................................. 110 Object Tagging ....................................................................................................................... 112 API Operations Related to Object Tagging ......................................................................... 113 Object Tagging and Additional Information ....................................................................... 114 Managing Object Tags ..................................................................................................... 118 Lifecycle Management ............................................................................................................. 121 When Should I Use Lifecycle Configuration for Objects? ...................................................... 121 How Do I Configure a Lifecycle? ....................................................................................... 122 Additional Considerations ................................................................................................ 122 Lifecycle Configuration Elements ...................................................................................... 127 Examples of Lifecycle Configuration .................................................................................. 133 Setting Lifecycle Configuration ......................................................................................... 143 CrossOrigin Resource Sharing (CORS) ....................................................................................... 151 CrossOrigin Resource Sharing: Usecase Scenarios .............................................................. 152 How Do I Configure CORS on My Bucket? .......................................................................... 152 How Does Amazon S3 Evaluate the CORS Configuration On a Bucket? ................................... 154 Enabling CORS ............................................................................................................... 154 Troubleshooting CORS .................................................................................................... 163 Operations on Objects ............................................................................................................ 164 Getting Objects .............................................................................................................. 164 Uploading Objects .......................................................................................................... 176 Copying Objects ............................................................................................................. 225 Listing Object Keys ......................................................................................................... 240 Deleting Objects ............................................................................................................. 247 Restoring Archived Objects .............................................................................................. 274 API Version 20060301 iv Amazon Simple Storage Service Developer Guide Storage Class Analysis ..................................................................................................................... 280 How to Set Up Storage Class Analysis ....................................................................................... 280 Storage Class Analysis ............................................................................................................. 281 How Can I Export Storage Class Analysis Data? .......................................................................... 283 Storage Class Analysis Export File Layout .......................................................................... 284 Amazon S3 Analytics REST APIs ............................................................................................... 284 Inventory ....................................................................................................................................... 286 How to Set Up Amazon S3 Inventory ........................................................................................ 286 Amazon S3 Inventory Buckets .......................................................................................... 286 Setting Up Amazon S3 Inventory ...................................................................................... 287 Inventory Lists ....................................................................................................................... 287 Inventory Consistency ..................................................................................................... 288 Location of Inventory Lists ...................................................................................................... 288 What is an Inventory Manifest? ........................................................................................ 289 Notify When Inventory Complete ............................................................................................. 290 Amazon S3 Inventory REST APIs .............................................................................................. 290 Managing Access ............................................................................................................................ 291 Introduction ........................................................................................................................... 291 Overview ....................................................................................................................... 292 How Amazon S3 Authorizes a Request .............................................................................. 297 Guidelines for Using the Available Access Policy Options ..................................................... 302 Example Walkthroughs: Managing Access .......................................................................... 305 Using Bucket Policies and User Policies ..................................................................................... 331 Access Policy Language Overview ..................................................................................... 331 Bucket Policy Examples ................................................................................................... 359 User Policy Examples ...................................................................................................... 368 Managing Access with ACLs ..................................................................................................... 390 Access Control List (ACL) Overview ................................................................................... 390 Managing ACLs ............................................................................................................... 395 Protecting Data .............................................................................................................................. 405 Data Encryption ..................................................................................................................... 405 ServerSide Encryption .................................................................................................... 406 ClientSide Encryption ..................................................................................................... 432 Reduced Redundancy Storage .................................................................................................. 441 Setting the Storage Class of an Object You Upload ............................................................. 442 Changing the Storage Class of an Object in Amazon S3 ....................................................... 443 Versioning ............................................................................................................................. 444 How to Configure Versioning on a Bucket .......................................................................... 445 MFA Delete .................................................................................................................... 446 Related Topics ................................................................................................................ 447 Examples ....................................................................................................................... 447 Managing Objects in a VersioningEnabled Bucket .............................................................. 449 Managing Objects in a VersioningSuspended Bucket .......................................................... 463 Hosting a Static Website ................................................................................................................. 467 Website Endpoints .................................................................................................................. 468 Key Differences Between the Amazon Website and the REST API Endpoint ............................. 469 Configuring a Bucket for Website Hosting ................................................................................. 469 Enabling Website Hosting ................................................................................................ 470 Configuring Index Document Support ............................................................................... 470 Permissions Required for Website Access ........................................................................... 472 (Optional) Configuring Web Traffic Logging ....................................................................... 472 (Optional) Custom Error Document Support ....................................................................... 473 (Optional) Configuring a Redirect ..................................................................................... 474 Example Walkthroughs ............................................................................................................ 481 Example: Setting up a Static Website ................................................................................ 481 Example: Setting up a Static Website Using a Custom Domain .............................................. 483 Example: Speed Up Your Website with Amazon CloudFront .................................................. 491 API Version 20060301 v Amazon Simple Storage Service Developer Guide Clean Up Example Resources ........................................................................................... 493 Notifications .................................................................................................................................. 495 Overview ............................................................................................................................... 495 How to Enable Event Notifications ............................................................................................ 496 Event Notification Types and Destinations ................................................................................. 498 Supported Event Types ................................................................................................... 498 Supported Destinations ................................................................................................... 499 Configuring Notifications with Object Key Name Filtering ............................................................ 499 Examples of Valid Notification Configurations with Object Key Name Filtering ........................ 500 Examples of Notification Configurations with Invalid PrefixSuffix Overlapping ....................... 502 Granting Permissions to Publish Event Notification Messages to a Destination ................................ 504 Granting Permissions to Invoke an AWS Lambda Function ................................................... 504 Granting Permissions to Publish Messages to an SNS Topic or an SQS Queue .......................... 504 Example Walkthrough 1 .......................................................................................................... 505 Walkthrough Summary ................................................................................................... 506 Step 1: Create an Amazon SNS Topic ................................................................................ 506 Step 2: Create an Amazon SQS Queue .............................................................................. 507 Step 3: Add a Notification Configuration to Your Bucket ...................................................... 508 Step 4: Test the Setup .................................................................................................... 511 Example Walkthrough 2 .......................................................................................................... 511 Event Message Structure ......................................................................................................... 511 CrossRegion Replication ................................................................................................................. 514 Usecase Scenarios .................................................................................................................. 514 Requirements ......................................................................................................................... 514 Related Topics ........................................................................................................................ 515 What Is and Is Not Replicated .................................................................................................. 515 What Is Replicated .......................................................................................................... 515 What Is Not Replicated ................................................................................................... 516 Related Topics ................................................................................................................ 517 How to Set Up ....................................................................................................................... 517 Create an IAM Role ......................................................................................................... 517 Add Replication Configuration .......................................................................................... 519 Walkthrough 1: Same AWS Account .................................................................................. 522 Walkthrough 2: Different AWS Accounts ............................................................................ 523 Using the Console .......................................................................................................... 527 Using the AWS SDK for Java ............................................................................................ 527 Using the AWS SDK for .NET ............................................................................................ 528 Replication Status Information ................................................................................................. 530 Related Topics ................................................................................................................ 532 Troubleshooting ..................................................................................................................... 532 Related Topics ................................................................................................................ 533 Replication and Other Bucket Configurations ............................................................................. 533 Lifecycle Configuration and Object Replicas ....................................................................... 533 Versioning Configuration and Replication Configuration ...................................................... 533 Logging Configuration and Replication Configuration .......................................................... 533 Related Topics ................................................................................................................ 533 Request Routing ............................................................................................................................. 534 Request Redirection and the REST API ...................................................................................... 534 Overview ....................................................................................................................... 534 DNS Routing .................................................................................................................. 534 Temporary Request Redirection ........................................................................................ 535 Permanent Request Redirection ........................................................................................ 537 DNS Considerations ................................................................................................................ 537 Performance Optimization ............................................................................................................... 539 Request Rate and Performance Considerations ........................................................................... 539 Workloads with a Mix of Request Types ............................................................................. 540 GETIntensive Workloads ................................................................................................. 542 API Version 20060301 vi Amazon Simple Storage Service Developer Guide TCP Window Scaling ............................................................................................................... 542 TCP Selective Acknowledgement .............................................................................................. 543 Monitoring ..................................................................................................................................... 544 Monitoring Tools .................................................................................................................... 544 Automated Tools ............................................................................................................ 544 Manual Tools ................................................................................................................. 545 Monitoring Metrics with CloudWatch ......................................................................................... 545 Metrics and Dimensions ................................................................................................... 546 Amazon S3 CloudWatch Daily Storage Metrics for Buckets ................................................... 546 Amazon S3 CloudWatch Request metrics ........................................................................... 546 Amazon S3 CloudWatch Dimensions ................................................................................. 548 Accessing CloudWatch Metrics .......................................................................................... 549 Related Resources ........................................................................................................... 550 Metrics Configurations for Buckets ............................................................................................ 550 BestEffort CloudWatch Metrics Delivery ............................................................................ 550 Filtering Metrics Configurations ........................................................................................ 551 How to Add Metrics Configurations ................................................................................... 551 Logging API Calls with AWS CloudTrail ...................................................................................... 552 Amazon S3 Information in CloudTrail ................................................................................ 552 Using CloudTrail Logs with Amazon S3 Server Access Logs and CloudWatch Logs .................... 556 Understanding Amazon S3 Log File Entries ........................................................................ 556 Related Resources ........................................................................................................... 558 BitTorrent ...................................................................................................................................... 559 How You are Charged for BitTorrent Delivery ............................................................................. 559 Using BitTorrent to Retrieve Objects Stored in Amazon S3 ........................................................... 560 Publishing Content Using Amazon S3 and BitTorrent .................................................................. 561 Amazon DevPay ............................................................................................................................. 562 Amazon S3 Customer Data Isolation ......................................................................................... 562 Example ........................................................................................................................ 563 Amazon DevPay Token Mechanism ........................................................................................... 563 Amazon S3 and Amazon DevPay Authentication ......................................................................... 563 Amazon S3 Bucket Limitation .................................................................................................. 564 Amazon S3 and Amazon DevPay Process ................................................................................... 565 Additional Information ............................................................................................................ 565 Error Handling ............................................................................................................................... 566 The REST Error Response ........................................................................................................ 566 Response Headers .......................................................................................................... 567 Error Response ............................................................................................................... 567 The SOAP Error Response ........................................................................................................ 568 Amazon S3 Error Best Practices ................................................................................................ 568 Retry InternalErrors ........................................................................................................ 568 Tune Application for Repeated SlowDown errors ................................................................ 568 Isolate Errors ................................................................................................................. 569 Troubleshooting Amazon S3 ............................................................................................................ 570 Troubleshooting Amazon S3 by Symptom ................................................................................. 570 Significant Increases in HTTP 503 Responses to Requests to Buckets with Versioning Enabled .... 570 Unexpected Behavior When Accessing Buckets Set with CORS .............................................. 571 Getting Amazon S3 Request IDs for AWS Support ...................................................................... 571 Using HTTP to Obtain Request IDs ................................................................................... 571 Using a Web Browser to Obtain Request IDs ...................................................................... 571 Using AWS SDKs to Obtain Request IDs ............................................................................. 572 Using the AWS CLI to Obtain Request IDs .......................................................................... 573 Related Topics ........................................................................................................................ 573 Server Access Logging ..................................................................................................................... 574 Overview ............................................................................................................................... 574 Log Object Key Format ................................................................................................... 575 How are Logs Delivered? ................................................................................................. 575 API Version 20060301 vii Amazon Simple Storage Service Developer Guide Best Effort Server Log Delivery ........................................................................................ 575 Bucket Logging Status Changes Take Effect Over Time ........................................................ 576 Related Topics ........................................................................................................................ 576 Enabling Logging Using the Console ......................................................................................... 576 Enabling Logging Programmatically .......................................................................................... 576 Enabling logging ............................................................................................................ 577 Granting the Log Delivery Group WRITE and READ_ACP Permissions ..................................... 577 Example: AWS SDK for .NET ............................................................................................. 578 Log Format ............................................................................................................................ 579 Custom Access Log Information ........................................................................................ 583 Programming Considerations for Extensible Server Access Log Format ................................... 583 Additional Logging for Copy Operations ............................................................................ 583 Deleting Log Files ................................................................................................................... 586 AWS SDKs and Explorers ................................................................................................................. 587 Specifying Signature Version in Request Authentication ............................................................... 588 Set Up the AWS CLI ................................................................................................................ 589 Using the AWS SDK for Java .................................................................................................... 590 The Java API Organization ............................................................................................... 591 Testing the Java Code Examples ....................................................................................... 591 Using the AWS SDK for .NET .................................................................................................... 591 The .NET API Organization ............................................................................................... 592 Running the Amazon S3 .NET Code Examples .................................................................... 593 Using the AWS SDK for PHP and Running PHP Examples ............................................................. 593 AWS SDK for PHP Levels ................................................................................................ 593 Running PHP Examples ................................................................................................... 594 Related Resources ........................................................................................................... 594 Using the AWS SDK for Ruby Version 2 ................................................................................... 595 The Ruby API Organization .............................................................................................. 595 Testing the Ruby Script Examples ..................................................................................... 595 Using the AWS SDK for Python (Boto) ....................................................................................... 596 Appendices .................................................................................................................................... 597 Appendix A: Using the SOAP API .............................................................................................. 597 Common SOAP API Elements ........................................................................................... 597 Authenticating SOAP Requests ......................................................................................... 598 Setting Access Policy with SOAP ....................................................................................... 599 Appendix B: Authenticating Requests (AWS Signature Version 2) ................................................... 600 Authenticating Requests Using the REST API ...................................................................... 601 Signing and Authenticating REST Requests ........................................................................ 603 BrowserBased Uploads Using POST ................................................................................. 612 Resources ...................................................................................................................................... 627 Document History .......................................................................................................................... 629 AWS Glossary ................................................................................................................................. 641