Dynamo Data Transform is an easy to use data transformation tool for DynamoDB
Dynamo Data Transform is an easy to use data transformation tool for DynamoDB.
It allows performing powerful data transformations using simple Javascript commands, without the risk of breaking your database. Available as a Serverless plugin, npm package and even as an interactive CLI, Dynamo Data Transform saves you time and keeps you safe with features like dry-running a data transformation and even rolling back your last trasnformation if needed.
Features
npm install dynamo-data-transform --save-dev
npx serverless plugin install -n dynamo-data-transform
Or add manually to your serverless.yml:
plugins:
- dynamo-data-transform
sls dynamodt --help
npm install -g dynamo-data-transform -s
dynamodt help
Or with the shortcut
ddt help
After installing the npm package, run:
dynamodt -i
sls dynamodt init --stage <stage>
Standalone:
ddt init --tableNames <table_names>
Open the generated data transformation file 'v1_script-name.js' file and implement the following functions:
dynamodt up
dynamodt down -t <table>
dynamodt prepare -t <table> --tNumber <transformation_number>
The function parameters:
dynamodt prepare
, you can use it here.dynamodt up
Make sure your script name contains the transformation number, for example: v1_transformation_script
const { utils } = require('dynamo-data-transform')
const TABLE_NAME = 'UsersExample'
const transformUp = async ({ ddb, isDryRun, preparationData }) => {
// your code here...
// return { transformed: 50 } // return the number of transformed items
}
const transformDown = async ({ ddb, isDryRun, preparationData }) => {
// your code here...
// return { transformed: 50 } // return the number of transformed items
}
const prepare = async ({ ddb, isDryRun }) => {
// your code here...
// return { transformed: 50 } // return the number of transformed items
}
module.exports = {
transformUp,
transformDown,
prepare, // optional
transformationNumber: 1,
}
List available commands: Serverless plugin:
sls dynamodt --help
Standalone npm package:
dynamodt help
To list all of the options for a specific command run: Serverless plugin:
sls dynamodt <command> --help
Standalone npm package:
dynamodt <command> --help
Examples of data transformation code
// Seed users data transformation
const { utils } = require('dynamo-data-transform');
const { USERS_DATA } = require('../../usersData');
const TABLE_NAME = 'UsersExample';
/**
* @param {DynamoDBDocumentClient} ddb - dynamo db document client https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb
* @param {boolean} isDryRun - true if this is a dry run
*/
const transformUp = async ({ ddb, isDryRun }) => {
return utils.insertItems(ddb, TABLE_NAME, USERS_DATA, isDryRun);
};
const transformDown = async ({ ddb, isDryRun }) => {
return utils.deleteItems(ddb, TABLE_NAME, USERS_DATA, isDryRun);
};
module.exports = {
transformUp,
transformDown,
transformationNumber: 1,
};
// Adding a "randomNumber" field to each item
const { utils } = require('dynamo-data-transform');
const TABLE_NAME = 'UsersExample';
const transformUp = async ({ ddb, isDryRun }) => {
const addRandomNumberField = (item) => {
const updatedItem = { ...item, randomNumber: Math.random() };
return updatedItem;
};
return utils.transformItems(ddb, TABLE_NAME, addRandomNumberField, isDryRun);
};
const transformDown = async ({ ddb, isDryRun }) => {
const removeRandomNumberField = (item) => {
const { randomNumber, ...oldItem } = item;
return oldItem;
};
return utils.transformItems(ddb, TABLE_NAME, removeRandomNumberField, isDryRun);
};
module.exports = {
transformUp,
transformDown,
transformationNumber: 2,
};
For more examples of data transformation code, see the examples folder in the repository.
Read how to automate data-transformation process here: https://www.infoq.com/articles/dynamoDB-data-transformation-safety/