Overview

Transformation Library is a powerful feature within the MetaRouter platform that enables users to create and utilize Lua Modules for transforming, filtering, and mapping data within Integrations and Pipelines. This layer provides reusable library code that can be applied across various stages, such as Pipelines and Integration Stages, offering a versatile and customizable approach to data handling. Currently, this tooling is only available via the CLI, but it will evolve into the UI as we learn more about its functionality.

Prerequisites

To effectively use the Transformation Library, ensure you have:

Access to the MetaRouter CLI.
Proficiency in Lua scripting and understanding of target endpoint schemas.
Knowledge of the data pipelines and events you intend to transform.
Necessary credentials for the systems being integrated.
Have basic familiarity of Modules Tutorial

Configuring Lua Modules within Transformation Library

With the necessary information and skills on hand, you can begin leveraging the Lua Modules in Transformation Library to transform, map and manipulate data across the MetaRouter data lifecycle:

Workflow

Access the MetaRouter CLI

Define Lua Modules

Create reusable Lua Modules by defining them in YAML files. For example:

yamlCopy code
name: mymodule
description: Fun times with Modules!
body: |
  local mymodule = {}

  function mymodule.add(a, b)
    return a + b
  end
  mymodule.HELLO = "HELLO LUA MODULE!!!!!!"
  return mymodule

Create Lua Modules in CLI

To create a Lua Module in the Control CLI, use the following command:
```
bashCopy code
control create lua-module -f <path_to_file>.yaml
```

A file can contain multiple Lua Modules using a -- document separator:

yamlCopy code
name: module1
description: Module 1
body: |
  local module1 = {}
  return module1

---

name: module2
description: Module 2
body: |
  local module2 = {}
  return module2

Apply Lua Module Functions

Use the defined modules in Pipeline Filters, Pipeline Transforms, Integration Transformation Stages, or any other location Lua is used in the platform. For example:

yamlCopy code
writeKey: test
transform:
  name: "Pipeline Transform"
  desc: "A Transformation that runs before integration transforms"
  data:
    playbook:
      version: v0.1.0
      global:
        expressions:
          - lang: lua
            body: |
              local mymodule = require "mymodule"
              input.properties.total = mymodule.add(input.properties.value1, input.properties.value2)
              input.properties.hello = mymodule.HELLO
              return input
integrations:
  - connection:

Example Usage

Given a test event:

jsonCopy code
{
  "event": "order_processed",
  "userId": "abc123",
  "writeKey": "example",
  "properties": {
    "value1": 10,
    "value2": 7
  }
}

Using the above module in a Pipeline Transform, the output would become:

jsonCopy code
{
  "event": "order_processed",
  "userId": "abc123",
  "writeKey": "example",
  "properties": {
    "value1": 10,
    "value2": 7,
    "total": 17,
    "hello": "HELLO LUA MODULE!!!!!!"
  }
}

Testing Transformation Library Lua Modules using Forge

Download the CLI Tool and create a local directory with the following structure. Make sure you create the lua-modules/section in order to test your Lua Modules

Required Forge Directory Structure

./
	tmp/
		forge.yaml
		forge.log
	pipeline-1/
		kit-1/
			connections.yaml
			playbook.yaml
			params.yaml (optional)
		kit-2/
			...
		...
	lua-modules/ (optional - directory name specified in forge.yaml)
		module1.yaml 
		module2.yaml
	fixtures/
		test.json

`tmp` Directory

This directory/file folder contains information used by the Forge tool to know what files to use for your testing environment. For Lua Modules, the LuaModulesDir: "./lua-modules"will be required in order to run successful tests.

forge.yaml - Provides the pathways you wish to use with Forge. Below is an example using the structure set within these instructions, not required to be exactly so.
```
DownloadTemplate: ""
InputFile: "./fixtures/test.json"
PipelineDir: "./pipeline"
KitDir: "./pipeline/kit-1"
LuaModulesDir: "./lua-modules"
```

`lua-modules` Directory

The name of the directory can be changed, LuaModulesDir will need to be updated in forge.yaml. Multiple Lua Module files can be loaded into this directory, they must end with a .yaml and a single file can also contain multiple modules containing the --- document separator. IE:

name: moda
body: |
  local moda = {}
  return moda
       
---
       
name: modb
body: |
  local modb = {}
  return modb

How Transformation Library Lua Modules Works

Lua Modules are designed to validate and apply transformations on input event data into MetaRouter . Here's an in-depth look at its functionality:

Comprehensive Playbook Processing:

The playbook includes multiple data transformation functions such as mappings, enrichments, spreading, custom expressions, and filters. When an input event is processed, a pipeline transformation or integration playbook can use a Lua module to aid in transforming events in a repeatable fashion.”

Data Function Execution:

Mappings: Translates specific event properties to the designated fields according to the playbook rules.
Enrichments: Integrates additional contextual data into the event, enhancing its informational value.
Spreading: Redistributes the event's data across various properties or formats.
Custom Expressions: Applies user-defined logic for tailored data transformations and computations.
Filters: Deleting and/or obfuscating event data that is collected by MetaRouter.

Real-Time Validation and Transformation:

Dynamically applies the configured transformations to produce the final data sent to downstream tools once the integration is deployed.

Benefits of Lua Modules within Transformation Library

Lua Modules addresses several critical issues in data transformation and management:

Efficiency: Reusable Lua Modules streamline workflows, reducing time and effort for data transformations.
Flexibility: Custom transformations, filters, and mappings tailored to specific requirements.
Control: Enhances accuracy and reliability of data integration processes.
Scalability: Supports complex data handling needs, managing large-scale data transformations and integrations.