What Is Google Pub/Sub?

Google Pub/Sub is a fully managed messaging service that facilitates asynchronous communication between distributed systems. It uses a publish/subscribe model, where producers send messages to a topic, and subscribers consume those messages. Pub/Sub is designed for high-throughput, low-latency message delivery, making it suitable for real-time event-driven architectures, data pipelines, and streaming analytics. It supports features such as message filtering, dead-letter topics, and at-least-once delivery, ensuring reliability and fault tolerance. Seamless integration with Google Cloud services enhances its utility for building scalable, resilient applications.

What are the benefits of integrating MetaRouter with Pub/Sub?

• Real-time Data Streaming: Enables real-time delivery of event data from MetaRouter to Google Pub/Sub for immediate processing.
• Scalable Architecture: Supports high-throughput data publishing, allowing the system to handle large volumes of event data efficiently.
• Seamless Integration: Works natively with Google Cloud services, simplifying downstream analytics, processing, and storage.

Capabilities and Considerations

• Event Publishing: The integration publishes all collected event data to a specified Google Pub/Sub topic for further processing.
• Global Expressions: Supports global transformations or enrichments via Lua expressions in the playbook configuration.
• Google Cloud Integration: Fully compatible with other Google Cloud services, such as BigQuery and Dataflow, for advanced data handling.
• Initial Setup Complexity: Configuration of credentials, project IDs, and topic names must be precise to avoid integration errors.

Limitations

• Single Topic: Event data is published to a single, pre-defined topic.
• No Built-in Filtering: The integration itself does not perform event filtering; filtering must be implemented downstream or through Pub/Sub's native features.

Getting Started - From Your Google Pub/Sub Account

1. Gathering Google Cloud Project Credentials

• Create a Service Account:
  1. Go to the Google Cloud Console.
  2. Navigate to IAM & Admin > Service Accounts.
  3. Click Create Service Account.
  4. Enter a name, description, and click Create and Continue.
  5. Assign the Pub/Sub Admin role to the service account for required permissions.
  6. Click Done to complete the service account setup.
• Download the JSON Key:
  1. Locate the newly created service account in the list.
  2. Click the three-dot menu next to the account and select Manage Keys.
  3. Click Add Key > Create New Key, and select the JSON format.
  4. Download the JSON file. This file contains the credentials needed for the credentials field in the YAML.
• Secure the Credentials:
  Store the JSON file securely and provide its path or contents as the credentials in the YAML configuration.
• For detailed instructions, refer to:
  • Creating and managing service accounts
  • Creating and managing service account keys

2. Finding the Project ID

• In the Google Cloud Console:
  1. Navigate to your project by clicking on the dropdown menu in the top navigation bar.
  2. The Project ID is listed next to the project name.
• Copy the Project ID and use it for the projectID field in the YAML.
• For more information, see:
  • Locate the project ID
• In addition, you can find the Project ID in your downloaded credentials JSON file, listed under the project_id field.

3. Creating and Identifying the Pub/Sub Topic

• Create a New Topic (if not already created):
  1. Go to Pub/Sub in the Google Cloud Console.
  2. Click Create Topic.
  3. Enter a topic name and click Create.
• Find the Topic Name:
  1. In the Pub/Sub section, locate your topic in the list.
  2. Copy the full topic name (e.g., projects/my-project-id/topics/my-topic).
  • Extract only the topic name (e.g., my-topic) for the topic field in the playbook configuration.
  • For detailed guidance, refer to:
    • Creating a topic

Getting Started - From Your MetaRouter Account

Adding a Google Pub/Sub integration.

From the integration library, add a Google Pub/Sub integration. Then, fill out the Connection Parameters:

Event Mappings

By default, the playbook includes a global Lua expression:

lua
CopyEdit
return input

This configuration ensures that all events are forwarded to Pub/Sub exactly as received, without modifications.

You can customize this behavior by modifying the Lua expression to transform event data as needed. Additionally, event filtering can be applied within the playbook configuration, similar to other MetaRouter integrations.

All events are published to the single Pub/Sub topic specified in the TOPIC parameter, and any additional routing or filtering must be handled downstream by Pub/Sub subscribers.

Additional Google Pub/Sub Documentation

• Google Pub/Sub Product Overview
• Creating and managing service accounts
• Creating and managing service account keys
• Locate the project ID
• Creating a topic