Edit me

MetaRouter makes it easy to send your data to an Amazon S3 bucket. Once you follow the steps below, your data will be routed through our platform and pushed to Amazon S3 in the appropriate format.

What is Amazon S3 and how does it work?

Amazon Simple Storage Service (S3) is a scalable web-based cloud storage service that allows you to store and retrieve data from anywhere at anytime. It offers an extremely flexible set of storage management and administration capabilities, supporting:

  • SSL transfers and automatic encryptions for guaranteed data security
  • Network-optimized, physical disk-based, or third-party connector methods for easy data imports and exports
  • Object tagging for easy category customization

Getting data into your S3 bucket natively requires quite a bit of developer work; you’d need to use an analytics tool like Google Analytics and write an ETL pipeline to bring the data from that analytics tool to S3. This is ties up your engineering resources and pre-formats the data in your S3 bucket, rather than giving you access to your raw data.

Why send data to Amazon S3 using MetaRouter?

We make delivering data to Amazon S3 possible with a few clicks, in the standardized analytics event format you are already using for other integrations.

When you enable the Amazon S3 integration, we begin delivering the raw event data you need to power your reports and custom analysis directly into your designated Amazon S3 bucket. Everything is stored as line-separated JSON objects, each containing one analytic event in its original format.

MetaRouter spares you the headache of writing custom ETL pipelines to get your data into buckets, and makes it easy to start your data engineering journey with s3.

Getting Started with Amazon S3 and MetaRouter

MetaRouter makes it easy to copy broadcasted events to an Amazon Simple Storage Service (S3) bucket. You’ll only need to configure the S3 Event Logs connector within your MetaRouter dashboard, and include our bucket policy.

Amazon S3 Side

Create an Amazon S3 bucket within your AWS account.

s3-event-logs1

Note: See below to confirm your region.

Add Permission for the Bucket

Add an S3 bucket policy, which will grant the IAM user permission to copy events into your S3 bucket using s3:PutObject. The following is an example of what the policy might look like.

{
    "Version": "2012-10-17",
    "Id": "Policy1475027027703",
    "Statement": [
        {
            "Sid": "Stmt1475027024994",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::213824691356:user/s3-copy"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        },
        {
            "Sid": "MetaRouterCopyAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::325517132571:user/s3-copy"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

s3-event-logs1

Please note the Access key and Secret Key for the AWS user. It will be used in the next section for configuration

MetaRouter Side

You can configure your integrations using a integrations.yaml config file, where you define multiple destinations, each one with its own custom settings.

The configuration needed for integrations is as follows:
bucket: The name of the bucket for which to save the events
region: Region where the bucket is located
secretKey: Secret key for the user to load data into the bucket
accessKey: Access key for the user to load data into the bucket
encryption (optional): the type of encryption to use in the bucket. Valid values are ‘AES256’ or ‘aws:kms’. If omitted the loader will store data without encryption

Here is an example of integrations.yaml:

- 	name: "s3"
    config:
        bucket: "test-bucket"
        region: "us-east-1"
        secretKey: "enter secret key here"
        accessKey: "enter access key here"
        encryption: "AES-256"

Things to Note:

  1. AWS Region (required) The region in which your S3 bucket resides. You can find a list of regions for Amazon S3 under the “Region” column here. Enter your region exactly as it’s shown (i.e. us-east-1).

  2. Server Side Encryption (optional) AWS S3 supports server side encryption. If your bucket policy requires the server side encryption header to be set, you can specify that here. Example values for this field are ‘AES256’ or ‘aws:kms’.

  3. Folder Structure Events are placed inside the bucket with the following folder structure Year/Month/Day/hour/events.json all events are loaded into files and chucked into smaller files by the forwarder