What Is Amazon S3?

Amazon S3 (Simple Storage Service) is a scalable, secure, and highly durable object storage service provided by AWS. It allows users to store and retrieve any amount of data, from anywhere on the web, making it ideal for a wide range of use cases such as backup, data archiving, content delivery, and big data analytics. With features like versioning, encryption, and lifecycle policies, S3 ensures data security and management flexibility. It also integrates seamlessly with other AWS services, enabling powerful workflows and applications.

---

Product Type: Storage
Integration Type: Starter Kit
Event Source Type: Web, Mobile Browser, and Mobile App
Event Scope: Full-Funnel Events

---

Capabilities

• Writes and manages customer data events in an Amazon S3 bucket.
• Supports specifying the AWS region to optimize performance and comply with data regulations.
• Enables global input handling and transformation using Lua scripting in the playbook.
• Supports customizable workflows for data processing and ingestion through playbook configuration.

Considerations

• Authentication uses an AWS Access Key and Secret Key. Ensure accessKey and secretKey are securely stored and not hard-coded in publicly accessible locations; use tools like AWS Secrets Manager for secure management.
• The integration operates against the specified AWS region. Choosing a region close to you helps minimize latency and cost.

Limitations

• The integration only works for the specified AWS region; misconfiguration of the region may lead to errors or increased latency.
• Any changes to the bucket name or credentials require manual updates to the configuration files.

---

Starter Kit Setup Guide

1. Gather Credentials

S3 Setup Guide

1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. In the left navigation pane, choose Buckets.
3. Choose Create bucket.
   1. The Create bucket page opens.
4. For Bucket name, enter a name for your bucket. The bucket name must:
   1. Be unique within a partition. A partition is a grouping of Regions. AWS currently has three partitions: aws (Standard Regions), aws-cn (China Regions), and aws-us-gov (AWS GovCloud (US) Regions).
   2. Be between 3 and 63 characters long.
   3. Consist only of lowercase letters, numbers, dots (.), and hyphens (-). For best compatibility, we recommend that you avoid using dots (.) in bucket names, except for buckets that are used only for static website hosting.
   4. Begin and end with a letter or number.
5. For Region, choose the AWS Region where you want the bucket to reside. To minimize latency and costs and address regulatory requirements, choose a Region close to you. Objects stored in a Region never leave that Region unless you explicitly transfer them to another Region. For a list of Amazon S3 AWS Regions, see AWS service endpoints in the Amazon Web Services General Reference.
6. Under Object Ownership, to disable or enable ACLs and control ownership of objects uploaded in your bucket, choose one of the following settings: ACLs disabled
   1. Bucket owner enforced (default) – ACLs are disabled, and the bucket owner automatically owns and has full control over every object in the bucket. ACLs no longer affect access permissions to data in the S3 bucket. The bucket uses policies exclusively to define access control. By default, ACLs are disabled. A majority of modern use cases in Amazon S3 no longer require the use of ACLs. We recommend that you keep ACLs disabled, except in unusual circumstances where you must control access for each object individually.
   2. Bucket owner preferred – The bucket owner owns and has full control over new objects that other accounts write to the bucket with the bucket-owner-full-control canned ACL. If you apply the Bucket owner preferred setting, to require all Amazon S3 uploads to include the bucket-owner-full-control canned ACL, you can add a bucket policy that allows only object uploads that use this ACL.
   3. Object writer – The AWS account that uploads an object owns the object, has full control over it, and can grant other users access to it through ACLs.
7. Under Block Public Access settings for this bucket, choose the Block Public Access settings that you want to apply to the bucket. By default, all four Block Public Access settings are enabled. We recommend that you keep all settings enabled, unless you know that you need to turn off one or more of them for your specific use case.
8. Under Default encryption, choose Edit.
9. To configure default encryption, under Encryption type, choose one of the following:
   1. Amazon S3 managed key (SSE-S3)
   2. AWS Key Management Service key (SSE-KMS)
10. Choose Create bucket.

How to Get Your ACCESS_KEY and SECRET_KEY

1. Log in to the AWS Management Console
   • Go to AWS Management Console at https://aws.amazon.com/console/ and log in with your credentials.
2. Navigate to the IAM (Identity and Access Management) Service
   • Search for IAM in the search bar and click on the IAM service.
3. Create or Use an Existing User
   • If you already have a user with programmatic access, skip to Step 5.
   • To create a new user:
     1. Click Users in the left-hand menu.
     2. Click the Add Users button.
     3. Enter a username and select Programmatic Access under Access Type.
4. Set User Permissions
   • Attach policies to the user that grant access to the S3 bucket. Common policies include:
     • AmazonS3FullAccess (grants full access to S3) or
     • A custom policy scoped to your specific bucket.
5. Get Your Access Key and Secret Key
   • Once the user is created, you will see an Access Key ID (this is your ACCESS_KEY) and Secret Access Key.
   • IMPORTANT: Save the SECRET_KEY securely as it will only be shown once. Use tools like AWS Secrets Manager or a password manager for storage.

2. Add an Amazon S3 Integration

• From the integration library, add an Amazon S3 integration. Then, fill out the Connection Parameters:

3. Configure Event Mapping

• MetaRouter provides all of the event mappings that Amazon S3 integrations typically require. You may add custom events, parameters, or mappings in accordance with Amazon S3's API documentation.

4. Deploy to Pipeline

• In the Pipelines tab, add your Amazon S3 integration.
• Select the correct integration revision.
• Click Add Integration to finalize deployment.

---

Event Mappings

Global

Global mappings will be applied to all events. If your parameter names do not match the Expected Inputs provided, you will need to overwrite the Inputs provided with your own.

---

Integration Validation

• In the AWS Management Console, open the Amazon S3 console and navigate to your bucket to confirm that event files are being written. If a prefix is configured, check inside that directory.
• Confirm the arrival of new event files after sending test traffic through the pipeline, and open a file to verify the event payload is stored as expected.
• If no files appear, verify your access key, secret key, bucket, and region, and contact your MetaRouter representative if the issue persists.

---

Additional Amazon S3 Documentation

• Managing access keys for IAM users
• AWS IAM User Guide
• Create your first S3 bucket

---