Transformation Library
Overview
Transformation Library is a powerful feature within the MetaRouter platform that enables users to create and utilize Lua Modules for transforming, filtering, and mapping data within Integrations and Pipelines. This layer provides reusable library code that can be applied across various stages, such as Pipelines and Integration Stages, offering a versatile and customizable approach to data handling. Currently, this tooling is only available via the CLI, but it will evolve into the UI as we learn more about its functionality.
Prerequisites
To effectively use the Transformation Library, ensure you have:
- Access to the MetaRouter CLI.
- Proficiency in Lua scripting and understanding of target endpoint schemas.
- Knowledge of the data pipelines and events you intend to transform.
- Necessary credentials for the systems being integrated.
- Have basic familiarity of Modules Tutorial
Configuring Lua Modules within Transformation Library
With the necessary information and skills on hand, you can begin leveraging the Lua Modules in Transformation Library to transform, map and manipulate data across the MetaRouter data lifecycle:
Workflow
-
Access the MetaRouter CLI
-
Define Lua Modules
- Create reusable Lua Modules by defining them in YAML files. For example:
yamlCopy code name: mymodule description: Fun times with Modules! body: | local mymodule = {} function mymodule.add(a, b) return a + b end mymodule.HELLO = "HELLO LUA MODULE!!!!!!" return mymodule
-
Create Lua Modules in CLI
-
To create a Lua Module in the Control CLI, use the following command:
bashCopy code control create lua-module -f <path_to_file>.yaml
-
A file can contain multiple Lua Modules using a
--
document separator:yamlCopy code name: module1 description: Module 1 body: | local module1 = {} return module1 --- name: module2 description: Module 2 body: | local module2 = {} return module2
- Apply Lua Module Functions
-
Use the defined modules in Pipeline Filters, Pipeline Transforms, Integration Transformation Stages, or any other location Lua is used in the platform. For example:
yamlCopy code writeKey: test transform: name: "Pipeline Transform" desc: "A Transformation that runs before integration transforms" data: playbook: version: v0.1.0 global: expressions: - lang: lua body: | local mymodule = require "mymodule" input.properties.total = mymodule.add(input.properties.value1, input.properties.value2) input.properties.hello = mymodule.HELLO return input integrations: - connection:
Example Usage
Given a test event:
jsonCopy code
{
"event": "order_processed",
"userId": "abc123",
"writeKey": "example",
"properties": {
"value1": 10,
"value2": 7
}
}
Using the above module in a Pipeline Transform, the output would become:
jsonCopy code
{
"event": "order_processed",
"userId": "abc123",
"writeKey": "example",
"properties": {
"value1": 10,
"value2": 7,
"total": 17,
"hello": "HELLO LUA MODULE!!!!!!"
}
}
Testing Transformation Library Lua Modules using Forge
Download the CLI Tool and create a local directory with the following structure. Make sure you create the lua-modules/
section in order to test your Lua Modules
Required Forge Directory Structure
./
tmp/
forge.yaml
forge.log
pipeline-1/
kit-1/
connections.yaml
playbook.yaml
params.yaml (optional)
kit-2/
...
...
lua-modules/ (optional - directory name specified in forge.yaml)
module1.yaml
module2.yaml
fixtures/
test.json
tmp
Directory
tmp
DirectoryThis directory/file folder contains information used by the Forge tool to know what files to use for your testing environment. For Lua Modules, the LuaModulesDir: "./lua-modules"
will be required in order to run successful tests.
forge.yaml
- Provides the pathways you wish to use with Forge. Below is an example using the structure set within these instructions, not required to be exactly so.DownloadTemplate: "" InputFile: "./fixtures/test.json" PipelineDir: "./pipeline" KitDir: "./pipeline/kit-1" LuaModulesDir: "./lua-modules"
lua-modules
Directory
lua-modules
DirectoryThe name of the directory can be changed, LuaModulesDir
will need to be updated in forge.yaml
. Multiple Lua Module files can be loaded into this directory, they must end with a .yaml
and a single file can also contain multiple modules containing the ---
document separator. IE:
name: moda
body: |
local moda = {}
return moda
---
name: modb
body: |
local modb = {}
return modb
How Transformation Library Lua Modules Works
Lua Modules are designed to validate and apply transformations on input event data into MetaRouter . Here's an in-depth look at its functionality:
Comprehensive Playbook Processing:
- The playbook includes multiple data transformation functions such as mappings, enrichments, spreading, custom expressions, and filters. When an input event is processed, a pipeline transformation or integration playbook can use a Lua module to aid in transforming events in a repeatable fashion.”
Data Function Execution:
- Mappings: Translates specific event properties to the designated fields according to the playbook rules.
- Enrichments: Integrates additional contextual data into the event, enhancing its informational value.
- Spreading: Redistributes the event's data across various properties or formats.
- Custom Expressions: Applies user-defined logic for tailored data transformations and computations.
- Filters: Deleting and/or obfuscating event data that is collected by MetaRouter.
Real-Time Validation and Transformation:
- Dynamically applies the configured transformations to produce the final data sent to downstream tools once the integration is deployed.
Benefits of Lua Modules within Transformation Library
Lua Modules addresses several critical issues in data transformation and management:
- Efficiency: Reusable Lua Modules streamline workflows, reducing time and effort for data transformations.
- Flexibility: Custom transformations, filters, and mappings tailored to specific requirements.
- Control: Enhances accuracy and reliability of data integration processes.
- Scalability: Supports complex data handling needs, managing large-scale data transformations and integrations.
Updated 25 days ago