What Is Amazon S3?

Amazon S3 (Simple Storage Service) is a scalable, secure, and highly durable object storage service provided by AWS. It allows users to store and retrieve any amount of data, from anywhere on the web, making it ideal for a wide range of use cases such as backup, data archiving, content delivery, and big data analytics. With features like versioning, encryption, and lifecycle policies, S3 ensures data security and management flexibility. It also integrates seamlessly with other AWS services, enabling powerful workflows and applications.

What are the benefits of integrating MetaRouter with Amazon S3?

• Customizable Data Workflows: The integration supports Lua scripting and playbook configurations, enabling tailored data transformations and workflows before storage.
• Secure and Scalable Connection: With AWS credentials and region-specific support, the integration ensures secure authentication and seamless scalability for handling large volumes of data.

Capabilities and Considerations

• Data Storage: Allows writing and managing data in an Amazon S3 bucket.
• Region-Specific Support: Supports specifying the AWS region to optimize performance and comply with data regulations.
• Global Data Handling: Enables global input handling and transformation using Lua scripting in the playbook.
• Customizable Workflows: Supports customizable workflows for data processing and ingestion through the playbook configuration.
• Credential Security: Ensure accessKey and secretKey are securely stored and not hard-coded in publicly accessible locations. Use tools like AWS Secrets Manager for secure management.

Limitations

• Region-Specific Operations: The integration only works for the specified AWS region; misconfiguration of the region may lead to errors or increased latency.
• Static Configuration: Any changes to the bucket name or credentials require manual updates to the configuration files.

Getting Started - From Your Amazon S3 Account

S3 Setup Guide

1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. In the left navigation pane, choose Buckets.
3. Choose Create bucket.
   1. The Create bucket page opens.
4. For Bucket name, enter a name for your bucket.The bucket name must:
   1. Be unique within a partition. A partition is a grouping of Regions. AWS currently has three partitions: aws (Standard Regions), aws-cn (China Regions), and aws-us-gov (AWS GovCloud (US) Regions).
   2. Be between 3 and 63 characters long.
   3. Consist only of lowercase letters, numbers, dots (.), and hyphens (-). For best compatibility, we recommend that you avoid using dots (.) in bucket names, except for buckets that are used only for static website hosting.
   4. Begin and end with a letter or number.
5. For Region, choose the AWS Region where you want the bucket to reside.To minimize latency and costs and address regulatory requirements, choose a Region close to you. Objects stored in a Region never leave that Region unless you explicitly transfer them to another Region. For a list of Amazon S3 AWS Regions, see AWS service endpoints in the Amazon Web Services General Reference.
6. Under Object Ownership, to disable or enable ACLs and control ownership of objects uploaded in your bucket, choose one of the following settings:ACLs disabled
   1. Bucket owner enforced (default) – ACLs are disabled, and the bucket owner automatically owns and has full control over every object in the bucket. ACLs no longer affect access permissions to data in the S3 bucket. The bucket uses policies exclusively to define access control.By default, ACLs are disabled. A majority of modern use cases in Amazon S3 no longer require the use of ACLs. We recommend that you keep ACLs disabled, except in unusual circumstances where you must control access for each object individually.
   2. Bucket owner preferred – The bucket owner owns and has full control over new objects that other accounts write to the bucket with the bucket-owner-full-control canned ACL.If you apply the Bucket owner preferred setting, to require all Amazon S3 uploads to include the bucket-owner-full-control canned ACL, you can add a bucket policy that allows only object uploads that use this ACL.
   3. Object writer – The AWS account that uploads an object owns the object, has full control over it, and can grant other users access to it through ACLs.
7. Under Block Public Access settings for this bucket, choose the Block Public Access settings that you want to apply to the bucket.By default, all four Block Public Access settings are enabled. We recommend that you keep all settings enabled, unless you know that you need to turn off one or more of them for your specific use case.
8. Under Default encryption, choose Edit.
9. To configure default encryption, under Encryption type, choose one of the following:
   1. Amazon S3 managed key (SSE-S3)
   2. AWS Key Management Service key (SSE-KMS)
10. Choose Create bucket.

How to Get Your ACCESS_KEY and SECRET_KEY

1. Log in to the AWS Management Console
   • Go to AWS Management Console and log in with your credentials.
2. Navigate to the IAM (Identity and Access Management) Service
   • Search for IAM in the search bar and click on the IAM service.
3. Create or Use an Existing User
   • If you already have a user with programmatic access, skip to Step 5.
   • To create a new user:
     1. Click Users in the left-hand menu.
     2. Click the Add Users button.
     3. Enter a username and select Programmatic Access under Access Type.
4. Set User Permissions
   • Attach policies to the user that grant access to the S3 bucket. Common policies include:
     • AmazonS3FullAccess (grants full access to S3) or
     • A custom policy scoped to your specific bucket.
5. Get Your Access Key and Secret Key
   • Once the user is created, you will see an Access Key ID (this is your ACCESS_KEY) and Secret Access Key.
   • IMPORTANT: Save the SECRET_KEY securely as it will only be shown once. Use tools like AWS Secrets Manager or a password manager for storage.

Getting Started - From Your MetaRouter Account

Adding an Amazon S3 integration.

From the integration library, add an Amazon S3 integration. Then, fill out the Connection Parameters:

Additional Amazon S3 Documentation

• Managing access keys for IAM users
• AWS IAM User Guide
• Create your first S3 bucket