📘
This integration is currently in beta. Contact your MetaRouter support team to get started.

What Is Databricks Zerobus?

Databricks Zerobus (Zerobus Ingest) is a streaming ingestion service that writes data directly into Databricks Unity Catalog Delta tables at high scale. It is a serverless offering that scales automatically without requiring teams to manage brokers, partitions, or pipelines. Data lands in Delta tables within seconds of arrival, making it query-ready for downstream analytics almost immediately. Zerobus is built to handle high-volume, concurrent writes from large numbers of producers into the same table.

Product Type: Data Warehouse
Integration Type: Starter Kit
Event Source Type: Web, Mobile Browser, & Mobile App
Event Scope: Full-Funnel Events

Capabilities

Streams full-funnel event data directly into a Unity Catalog Delta table via the Zerobus REST API.
Sends batches of events per request.
Writes core event fields as top-level columns and passes context, properties, and traits as structured values.
Authenticates using OAuth client credentials tied to a Databricks service principal.
Supports web, mobile browser, and mobile app event collection.

Considerations

A target Delta table must be created before events can be sent. The table's column schema is defined by MetaRouter, and each record's payload must match that schema.
Zerobus rejects any record containing a field that has no matching column in the target table. To leave out a column, omit it from the CREATE TABLE statement and delete the mapping of the same name in the kit. Both steps are required.
context, properties, and traits arrive as JSON objects with no fixed shape, so they are stored as VARIANT values that keep nested fields queryable. Columns with no value in a given event are written as null.
Authentication requires a Databricks service principal with a generated client ID and client secret. The service principal must be granted USE CATALOG, USE SCHEMA, and MODIFY plus SELECT privileges on the catalog, schema, and target table.
The ZEROBUS_ENDPOINT follows the format $$WORKSPACE_ID$$.zerobus.$$REGION$$.$$CLOUD_DOMAIN$$. The CLOUD_DOMAIN depends on the cloud the workspace runs on: cloud.databricks.com for AWS, azuredatabricks.net for Azure, and gcp.databricks.com for GCP.
Zerobus Ingest is available in select regions. The workspace and target table must both be in a supported region.
Zerobus Ingest guarantees ordering per stream, not globally across streams.
This integration uses the Zerobus REST API, which is in beta.

Limitations

The Zerobus Ingest limitations documentation is published separately per cloud, and the versions differ in content, not just in domain names. Latency figures, the partitioned-table limit, and the list of unsupported workspace types are not identical across AWS, Azure, and GCP. Review the version matching the customer's cloud rather than assuming the AWS page applies.

Starter Kit Setup Guide

1. Gather Credentials

Create a Databricks service principal and gather the connection values before configuring the integration.

Create a service principal:
1. Navigate to Settings > Identity and Access.
2. Under Service principals, select Manage, then click Add service principal.
3. In the Add service principal window, click Add new to create a new service principal.
4. Generate and save the client ID and client secret for the service principal.
Grant the service principal the required permissions on the catalog, schema, and table:
1. In the Service principal page, open the Configurations tab and copy the Application ID (UUID).
2. Grant USE CATALOG on the catalog, USE SCHEMA on the schema, and MODIFY plus SELECT on the target table to that UUID.
Gather the remaining values:
- WORKSPACE_URL: Everything before the /o=XXXXX in the URL shown when you log in (for example, https://abcd-teste2-test-spcse2.cloud.databricks.com).
- WORKSPACE_ID: The value after /o= in the workspace URL (for example, 2281745829657864).
- ZEROBUS_ENDPOINT: Formatted as $$WORKSPACE_ID$$.zerobus.$$REGION$$.$$CLOUD_DOMAIN$$. Find the region in the workspace switcher in the top navigation bar of the Databricks UI, or in the account console under Workspaces.
- CATALOG and SCHEMA: In the Databricks UI, click Catalog in the left sidebar. The top level of the tree is the catalog; expanding it shows the schemas inside.
- TABLE: The name of the target Delta table you create for this integration.

2. Add a Databricks Zerobus Integration

From the integration library, add a Databricks Zerobus integration. Then, fill out the Connection Parameters:

Connection Parameter	Description
`ZEROBUS_ENDPOINT`	Zerobus ingest endpoint host, formatted as workspace ID, region, and cloud domain.
`CLIENT_ID`	Service principal client ID used to request the OAuth access token.
`CLIENT_SECRET`	Service principal client secret used to request the OAuth access token.
`WORKSPACE_URL`	Base workspace URL used as the OAuth token endpoint host.
`WORKSPACE_ID`	Databricks workspace identifier used in the OAuth authorization request.
`CATALOG`	Unity Catalog catalog containing the target table.
`SCHEMA`	Schema within the catalog containing the target table.
`TABLE`	Target Delta table that events are written to.

3. Configure Event Mapping

MetaRouter provides all of the event mappings that Databricks Zerobus integrations typically require. You may add custom events, parameters, or mappings in accordance with Databricks Zerobus's API documentation.

4. Deploy to Pipeline

In the Pipelines tab, add your Databricks Zerobus integration.
Select the correct integration revision.
Click Add Integration to finalize deployment.

Event Mappings

Global

Global mappings will be applied to all events. If your parameter names do not match the Expected Inputs provided, you will need to overwrite the Inputs provided with your own.

Output Key	Description	Expected Input
`anonymousId`	String: Anonymous visitor identifier.	`anonymousId`
`userId`	String: Authenticated user identifier.	`userId`
`event`	String: Event name.	`event`
`type`	String: Event type.	`type`
`channel`	String: Channel the event originated from.	`channel`
`messageId`	String: Unique message identifier.	`messageId`
`timestamp`	String: Event timestamp.	`timestamp`
`originalTimestamp`	String: Original client-side event timestamp.	`originalTimestamp`
`sentAt`	String: Time the event was sent.	`sentAt`
`receivedAt`	String: Time the event was received.	`receivedAt`
`context`	Variant: Event context object stored as structured data.	Expression – serializes `context` to a scalar string
`properties`	Variant: Event properties object stored as structured data.	Expression – serializes `properties` to a scalar string
`traits`	Variant: User traits object stored as structured data.	Expression – serializes `traits` to a scalar string

Event Specific

All Events

Output Key	Description	Expected Input
`event`	String: Event name applied consistently across all event types.	`event`

Required & Recommended Identifiers

These identifiers must be mapped to Databricks Zerobus in order for successful user matching to occur. Without these IDs, any events sent to Databricks Zerobus may not be accurately reflected in reporting.

Attribute	Example	Sync Injector Required?
`anonymousId`	`b171a9b06ce011ecafcd1b209be8601b`	No
`userId`	`111222`	No

Integration Validation

The Zerobus Ingest REST API returns error codes that indicate whether a request succeeded and how a client should respond. Note that responses should be reviewed for errors even when a request appears to be accepted.

Client errors indicate a problem with the request and should not be retried without modification:

Error Code (REST)	HTTP Status	Description	Recommended Action
`INVALID_PARAMETER_VALUE`	400	The request contains invalid or malformed input, such as a missing required field, an invalid schema, or an unsupported record format.	Fix the request and resubmit. Inspect the `message` field to identify the invalid parameter.
`NOT_FOUND`	404	The requested resource does not exist, such as a target table that cannot be found.	Verify that the resource name is correct and that it exists.
`NOT_IMPLEMENTED`	501	The requested operation is not supported, such as a table using an unsupported feature or data format.	Do not retry. Check the `message` field to identify what is unsupported.

Authentication and authorization errors indicate an issue with the caller's identity or permissions and should not be retried with the same credentials:

Error Code (REST)	HTTP Status	Description	Recommended Action
`UNAUTHENTICATED`	401	The request does not have valid authentication credentials. The token may be missing, empty, expired, or invalid.	Refresh or provide a valid authentication token and retry.
`PERMISSION_DENIED`	403	The caller does not have sufficient privileges for the requested operation on the resource.	Verify that the caller has the required privileges (such as `MODIFY`, `SELECT`, `USE CATALOG`, `USE SCHEMA`) on the target resource.

Server errors indicate a problem on the server side and can be retried with exponential backoff and jitter:

Error Code (REST)	HTTP Status	Description	Recommended Action
`UNAVAILABLE`	503	The service is temporarily unable to handle the request. This is typically a transient condition.	Retry with exponential backoff and jitter.
`RESOURCE_EXHAUSTED`	429	The service is rejecting requests due to resource limits.	Reduce request concurrency if possible. Retry with exponential backoff and jitter.
`INTERNAL_ERROR`	500	An unexpected internal error occurred.	Do not retry. Contact support and provide the full error response for diagnosis.

The REST API supports up to 10,000 requests per second by default. To raise this limit, contact your Databricks account representative.

Additional Databricks Zerobus Documentation

Zerobus Ingest overview: https://docs.databricks.com/aws/en/ingestion/zerobus-overview
Use the Zerobus Ingest connector: https://docs.databricks.com/aws/en/ingestion/zerobus-ingest
Zerobus Ingest connector limitations: https://docs.databricks.com/aws/en/ingestion/zerobus-limits
Authorize service principal access with OAuth (M2M): https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m