Data Replay

Data Replay re-streams previously ingested, untransformed events from your cloud object
storage back through MetaRouter and out to your configured integrations. Use it when:

• An integration's API experiences downtime.
• A destination temporarily cannot accept events.
• Events were malformed during transformation.

Replayed events follow the same transformation and routing logic configured in your playbooks,
so you can replay data using updated configurations when the original run was misconfigured.

• Replay jobs are submitted through the Control CLI.
• Storage credentials are stored and referenced as secrets, see
  Managing Secrets.

How It Works

1. Submit a replay config with control create replay. MetaRouter validates it, snapshots
   it for the run, and starts one replay job.
2. Read. The job lists your bucket with the config's glob pattern and streams each
   matching file.
3. Filter. Events pass through the config's scopes. Filtered events are dropped before
   they enter the platform.
4. Deliver. Surviving events flow through MetaRouter in the same shape as live traffic,
   through the write key's current pipeline configuration, out to its integrations.

Prerequisites and Limitations

• Contract and enablement. Data Replay must be included in your MetaRouter contract and
  enabled for your cluster. Contact your MetaRouter representative before your first replay.
• Admin role. Submitting a replay requires the Admin role.
• Credentials are stored as secrets. Source storage credentials live in a secret, stored
  once and referenced by the replay config, never inline. See
  Managing Secrets.
• Amazon S3 only. Source events must be stored in S3. Google Cloud Storage and Azure Blob
  Storage are not yet supported, contact the MetaRouter team if you use another provider.
• Raw events. Files must hold the original events as first ingested, before any pipeline
  transforms or filters. Events that never reached the bucket cannot be replayed.
• Current pipeline config applies. Replay runs events through the pipeline as configured
  now, not as configured when the events first flowed. Confirm the pipeline is in the intended
  state before replaying. A misconfiguration still in place drops the events again.
• Duplicates are possible. Nothing dedups replayed events, and a failed run retries from
  the beginning. Downstream integrations must tolerate duplicates. There is no dry run.
• No ordering guarantee. Events are delivered in no particular order. Each keeps its
  original timestamp, so destinations that place events by time are unaffected.
• One replay at a time. Later submissions wait until the running one finishes.
• Runtime and size limits. A run times out after 2 hours, which comfortably covers 1 to 2
  million average-size events. Each event must be under 500 KB.

Setting Up Bucket Access

Step 1: Connect Your Bucket

• Create and configure an Amazon S3 bucket
  if one is not already set up.
• Connect MetaRouter to this bucket using the
  S3 integration guide.
• Ensure the bucket is integrated with the pipelines you want to replay events through.

Step 2: Grant AWS Permissions

Create a dedicated IAM user for replay and grant it read access to the bucket:

• s3:ListBucket on the bucket, so the glob pattern can discover files.
• s3:GetObject on the bucket's objects.
• kms:Decrypt on the bucket's key, only when objects are encrypted with a customer-managed
  KMS key.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReplayListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<your-bucket-name>"
    },
    {
      "Sid": "ReplayReadObjects",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::<your-bucket-name>/*"
    },
    {
      "Sid": "ReplayKMSDecrypt",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"
    }
  ]
}

Step 3: Store the Credentials as a Secret

Store the IAM user's access key pair as a secret. The replay config references it by ID and
never holds credentials inline. See
Managing Secrets.

control create secret -f replay-source.yaml

id: replay-source
description: AWS creds for replay bucket
values:
  AWS_ACCESS_KEY: <access-key-id>
  AWS_SECRET_KEY: <secret-access-key>

Step 4: Clean Up the Destination (If Needed)

If the events you intend to replay were malformed or incorrectly delivered, delete them from
the destination system first. This creates a clean window that the replay can refill without
duplication.

CLI Configuration

• A replay config has an id, an object-storage source, a globPattern for file discovery,
  and optional scopes for filtering and routing. The id must use write-key characters
  (alphanumeric, _, -).
• Start from the annotated example, do not write a config from scratch.
• Credentials are never inline. Credential fields hold $$secret.KEY$$ placeholders that
  resolve from the secret named by secretRef on sourceConfig.source.

control create replay -f <config.yaml>   # validate, snapshot, and submit a replay job
control create replay --example          # print the annotated example config

The create response carries the generated run_id and workflow_name for tracking the run.

Storage Sources

Glob and Scopes

• globPattern selects which object keys to replay. Required. Examples:
  • events/**/*.json.gz matches every gzipped NDJSON file under events/
  • replay-test/**.json.gz matches one prefix's exports
• scopes is optional. Omit it to replay every event in the matched files. It carries two
  mechanisms: three fields filter events out, two fields do not filter but restrict which
  integrations receive the events.

Config Example

Print the canonical version with control create replay --example.

id: mr-dev-replay-test
sourceConfig:
  source:
    s3:
      region: us-east-1
      bucket: mr-test-replay-us-east-1
      credentials:
        accessKey: $$secret.AWS_ACCESS_KEY$$
        secretKey: $$secret.AWS_SECRET_KEY$$
    secretRef: replay-source
  globPattern: replay-test/**.json.gz
  scopes:
    writeKeys:                   # keep only these write keys (empty/omit = all)
      - retl_replay
    events:                      # keep only these (normalized) event names (empty/omit = all)
      - identify
      - page
      - product_viewed
      - order_completed
      - signup_started
      - add_to_audience
    integrationIds:              # on ingest, deliver ONLY to these integration IDs (empty/omit = all)
      - 91aab3c7-4ae4-49f9-b051-2ab2bfcfdf50
    timeRange:                   # inclusive window, omit to ignore timestamps
      start: 2026-07-03T00:00:00Z   # required when timeRange is set
      end:   2026-07-05T23:59:59Z   # optional, open-ended if omitted, must be >= start

Source File Format

Replay reads newline-delimited JSON (NDJSON), one event per line, each line a valid JSON
object.

Do not wrap events in a JSON array or add trailing commas. Any path layout works, but for
large exports we recommend partitioning by time with Hive-style paths, for example
s3://bucket/prefix/year=2026/month=06/day=15/{uuid}.json.gz. A glob can then select a time
window directly, and replay only lists and reads the matching portion of the bucket.

Event Formats

Replay accepts two event shapes, and they can be mixed within a file.

Analytics.js format, what MetaRouter SDKs produce. Requires writeKey, type, and at
least one of userId or anonymousId:

{
  "writeKey": "your-write-key",
  "type": "track",
  "event": "Order Completed",
  "anonymousId": "550e8400-e29b-41d4-a716-446655440000",
  "userId": "user-123",
  "timestamp": "2026-01-15T14:30:00.000Z",
  "properties": { "orderId": "ORD-12345", "total": 99.99 }
}

Custom format with _metarouter, wrapping arbitrary JSON with a _metarouter metadata
block:

{
  "orderId": "ORD-12345",
  "_metarouter": {
    "writeKey": "your-write-key",
    "eventID": "evt-550e8400",
    "eventName": "Order Completed",
    "timestamp": "2026-01-15T14:30:00Z",
    "userID": "user-123"
  }
}

Required fields in _metarouter: writeKey, eventID, eventName, timestamp. Optional:
userID, anonymousID, ip.

Timestamps are ISO-8601, UTC preferred (2026-01-15T14:30:00Z).

FAQs

Can I replay from Google Cloud Storage or Azure Blob Storage?
Not yet. Amazon S3 is the only supported source today, with access keys or workload identity.
Contact the MetaRouter team if you use another provider.

How long can a replay run?
2 hours by default, then the job times out. That window comfortably covers 1 to 2 million
average-size events. The limit can be raised per cluster, contact MetaRouter if you need
longer.

What happens if events are missing timestamp or receivedAt?
Replay preserves each event's original time from receivedAt or timestamp. When neither is
present, the event is stamped with the replay time.

Can I re-query or delete a replay config?
No. Replay is fire-and-forget. Each create snapshots the config for that run and submits a
job. Track the run with the run_id and workflow_name in the create response.

How do I tell if the replay worked?
Check the downstream integration's vendor admin or analytics UI for the replayed events, that
is the definitive confirmation they were delivered. You can also watch events flow with the
event debugger in the MetaRouter UI. Keep in mind replayed events keep their original
timestamps, so depending on the timestamp set in the events they may appear in the past
rather than at the top of a recent view. If you do not see any events, contact MetaRouter to
check whether the replay run failed.

Are duplicates possible?
Yes. There is no checkpoint or dedup. A retry re-streams whole files and re-delivers their
events, so downstream integrations should tolerate duplicates.

What is the difference between filters and routing in scopes?
events, writeKeys, and timeRange drop events before they enter the platform.
integrations and integrationIds do not filter, they only restrict which integrations the
replayed events reach.

Do replayed events keep their original time?
Yes. Replay stamps each event's receivedAt from its own receivedAt or timestamp, falling
back to the replay time only when neither is present.

Where do the storage credentials live?
In the secret named by secretRef on sourceConfig.source, never inline. $$secret.KEY$$
placeholders resolve at runtime. See
Managing Secrets.