What Is Google Pub/Sub?

Google Pub/Sub is a fully managed messaging service that facilitates asynchronous communication between distributed systems. It uses a publish/subscribe model, where producers send messages to a topic, and subscribers consume those messages. Pub/Sub is designed for high-throughput, low-latency message delivery, making it suitable for real-time event-driven architectures, data pipelines, and streaming analytics. It supports features such as message filtering, dead-letter topics, and at-least-once delivery, ensuring reliability and fault tolerance. Seamless integration with Google Cloud services enhances its utility for building scalable, resilient applications.

What are the benefits of integrating MetaRouter with Pub/Sub?

• Real-time Data Streaming: Enables real-time delivery of event data from MetaRouter to Google Pub/Sub for immediate processing.
• Scalable Architecture: Supports high-throughput data publishing, allowing the system to handle large volumes of event data efficiently.
• Seamless Integration: Works natively with Google Cloud services, simplifying downstream analytics, processing, and storage.

Capabilities and Considerations

• Workload Identity: If your MetaRouter deployment runs on Google Kubernetes Engine (GKE), you can authenticate with Pub/Sub using Workload Identity instead of a JSON service account key. Workload Identity allows your MetaRouter pods to authenticate directly with Google Cloud APIs without managing static credentials, providing a more secure and maintenance-free authentication method. This is the recommended path.
  • Workload Identity is supported only for clusters running in Google Cloud (GKE). Deployments hosted outside of GCP must use the Service Account JSON key authentication method.
• Event Publishing: The integration publishes all collected event data to a specified Google Pub/Sub topic for further processing.
• Global Expressions: Supports global transformations or enrichments via Lua expressions in the playbook configuration.
• Google Cloud Integration: Fully compatible with other Google Cloud services, such as BigQuery and Dataflow, for advanced data handling.
• Initial Setup Complexity: Configuration of credentials, project IDs, and topic names must be precise to avoid integration errors.

Limitations

• Single Topic: Event data is published to a single, pre-defined topic.
• No Built-in Filtering: The integration itself does not perform event filtering; filtering must be implemented downstream or through Pub/Sub's native features.

Getting Started - From Your Google Pub/Sub Account

Step 1: Finding the Project ID

Finding the Project ID

• In the Google Cloud Console:
  1. Navigate to your project by clicking on the dropdown menu in the top navigation bar.
  2. The Project ID is listed next to the project name.
• Copy the Project ID and use it for the projectID field in the YAML.
• For more information, see:
  • Locate the project ID
• In addition, you can find the Project ID in your downloaded credentials JSON file, listed under the project_id field.

Step 2: Choose Your Authentication Method

Path A — Workload Identity Flow

• If you are using the Workload Identity flow, please reach out to your MetaRouter representative. We will generate a service account email for you.

• Assign the necessary roles to the service account email provided by the MetaRouter team. You will need to grant at least the "Pub/Sub Publisher" role.

• Leave the CREDENTIALS field empty in your MetaRouter Pub/Sub integration configuration. When no credentials are provided, the integration automatically falls back to GCP default credentials, which Workload Identity supplies.

Path B — Service Account Key Flow

• Create a Service Account:
  1. Go to the Google Cloud Console.
  2. Navigate to IAM & Admin > Service Accounts.
  3. Click Create Service Account.
  4. Enter a name, description, and click Create and Continue.
  5. You will need to grant at least the "Pub/Sub Publisher" role.
  6. Click Done to complete the service account setup.
• Download the JSON Key:
  1. Locate the newly created service account in the list.
  2. Click the three-dot menu next to the account and select Manage Keys.
  3. Click Add Key > Create New Key, and select the JSON format.
  4. Download the JSON file. This file contains the credentials needed for the credentials field in the YAML.
• Secure the Credentials: Store the JSON file securely and provide its path or contents as the credentials in the YAML configuration.
• For detailed instructions, refer to:
  • Creating and managing service accounts
  • Creating and managing service account keys

Step 3: Creating and Identifying the Pub/Sub Topic

• Create a New Topic (if not already created):
  1. Go to Pub/Sub in the Google Cloud Console.
  2. Click Create Topic.
  3. Enter a topic name and click Create.
• Find the Topic Name:
  1. In the Pub/Sub section, locate your topic in the list.
  2. Copy the full topic name (e.g., projects/my-project-id/topics/my-topic).
  • Extract only the topic name (e.g., my-topic) for the topic field in the playbook configuration.
  • For detailed guidance, refer to:
    • Creating a topic

Getting Started - From Your MetaRouter Account

Adding a Google Pub/Sub integration.

From the integration library, add a Google Pub/Sub integration. Then, fill out the Connection Parameters:

Event Mappings

By default, the playbook includes a global Lua expression:

lua
CopyEdit
return input

This configuration ensures that all events are forwarded to Pub/Sub exactly as received, without modifications.

You can customize this behavior by modifying the Lua expression to transform event data as needed. Additionally, event filtering can be applied within the playbook configuration, similar to other MetaRouter integrations.

All events are published to the single Pub/Sub topic specified in the TOPIC parameter, and any additional routing or filtering must be handled downstream by Pub/Sub subscribers.

Additional Google Pub/Sub Documentation

• Google Pub/Sub Product Overview
• Creating and managing service accounts
• Creating and managing service account keys
• Locate the project ID
• Creating a topic