Benerator, the smart way to generate data

rapiddweller-benerator-ce

Welcome to rapiddweller Benerator, a powerful software solution for data generation, obfuscation, and migration for development, testing, and training purposes. Our core goals are to provide model-driven test data generation that satisfies complex data validity requirements, anonymize production data for showcases and serious performance testing projects, efficiently generate large data volumes, and offer wide customization possibilities and ease of use for non-developers.

Quickstart ( use benerator version from releases )

1. make sure you have installed Java 11 JDK or higher and JAVA_HOME environment variable is set correctly

2. download the latest benerator version from Releases

3. unzip .tar.gz to choose/your/path

4. add environment variable BENERATOR_HOME=choose/your/path and add choose/your/path/bin to your PATH variable For example (these environment variables are only set in your terminal session, read more about environment variables here)

   • Linux bash

     export BENERATOR_HOME=/home/user1/rapiddweller-benerator-ce-3.1.0-jdk-11-dist  
     export PATH=$BENERATOR_HOME/bin:$PATH 
     chmod a+x $BENERATOR_HOME/bin/*.sh
     

   • Mac Install with Homebrew Link to Homebrew Formular here

     brew install benerator
     

   • Windows 10 PowerShell

     $env:BENERATOR_HOME='C:\Users\user1\rapiddweller-benerator-ce-3.1.0-jdk-11-dist'  
     $env:Path += 'C:\Users\user1\rapiddweller-benerator-ce-3.1.0-jdk-11-dist\bin'
     

5. create your own benerator script myscript.xml with the following content

   <setup>
       <import domains="person,organization"/>
       <generate type="customer" count="1000" threads="1" consumer="LoggingConsumer,CSVEntityExporter">
         <variable name="person" generator="new PersonGenerator{minAgeYears='21', maxAgeYears='67',femaleQuota='0.5'}" />
         <variable name="company" generator="CompanyNameGenerator" />
         <attribute name="first_name" script="person.familyName" />
         <attribute name="last_name" script="person.givenName" />
         <attribute name="birthDate" script="person.birthDate" converter="new java.text.SimpleDateFormat('dd.MM.YYYY')" />
         <attribute name="superuser" values="true, false" />
         <attribute name="salutation" script="person.salutation " />
         <attribute name="academicTitle" script="person.academicTitle" />
         <attribute name="email" script="'info@' + company.shortName.replace(' ', '-') + this.tc_creation + '.de'" />
       </generate>
     </setup>
   

6. run your first benerator script

   benerator myscript.xml
   

And that's it! You're ready to start using rapiddweller Benerator CE. We hope you find it as useful as we do. If you have any questions, please reach out to us on GitHub or visit our website www.benerator.de for more information.

Don't forget to leave us a star on GitHub if you found this project helpful!

Please note that these instructions are a basic guide to get you started with the software, and you may need to consult the documentation or seek support for more detailed usage instructions.

Introduction

rapiddweller Benerator is a model-driven test data generation tool that allows you to create realistic and valid high-volume test data for testing (unit/integration/load) and showcase setup. It offers several features such as:

• Generation of data that satisfies complex data validity requirements
• Anonymization of production data for showcases and serious performance testing projects
• Efficient generation of large data volumes, scaling up to companies with billions of customers and Big Data projects
• Early applicability in projects
• Efficiency by creating reusable and extendable data models
• Low Code approach to have more transparency
• Little maintenance effort with ongoing implementation through configuration by exception
• Wide and easy customization possibilities
• Applicability by non-developers
• Intuitive data definition format
• Satisfying stochastic requirements on data
• Extraction and anonymization of production data
• Supporting distributed and heterogeneous applications
• Establishing a common data generation platform for different business domains and software systems

Data Synthesis

Performance test data can be completely synthesized. A basic setup can be imported e.g. from DbUnit files, CSV files and fixed column width files. A descriptor file configures how imported data should be processed and adds completely synthesized data. The processed or generated data finally is stored in the system under test.

Production Data Anonymization

Production data can be easily extracted from production systems. Tables can be imported unmodified, filtered, anonymized and converted.

Building Blocks

Benerator is developed and continuously extended and improved since June 2006. Benerator is mainly used and tested best for the data file and database data generation, for these applications Benerator should help you with almost all your data generation needs out of the box - and extending Benerator for specific needs is easy.

rapiddweller Benerator is built for Java 11

If you need support for Java 8 or earlier, please consider using the versions <= 1.0.1.

Prerequisites

• Java 11 JDK (we recommend graalvm or azul)
• Maven

Check your local setup

java -version
mvn -version

Installation

1. Choose how to install:

   a) Download a Prebuilt Distribution from Project Overview > Releases (current release is 3.1.0, cp. rapiddweller-benerator-ce-3.1.0-jdk-11-dist.zip) and unzip the downloaded file in an appropriate directory, e.g. /Developer/Applications or C:\Program Files\Development.

   b) Checkout repository and build your own rapiddweller-benerator-ce using
   maven commands mvn clean install followed by mvn assembly:single (now you have a packed distribution af the current development)

   Please note: We highly recommend using option 1a and download our release packages to ease your start. If you clone our GitHub repository, there are no binaries included, and you need to build benerator yourself. Building benerator requires a proper java/maven setup on your system. Learn more in Chapter Prerequisites.

2. Set BENERATOR_HOME
   Create an environment variable BENERATOR_HOME that points to the path you extracted Benerator to.

   • Windows Details: Open the System Control Panel, choose Advanced Settings - Environment Variables. Choose New in the User Variables section. Enter BENERATOR_HOME as name and the path as value (e.g. C:\Program Files\Development\rapiddweller-benerator-ce-3.1.0-jdk-11). Click OK several times.

   • Mac/Unix/Linux Details: Add an entry that points to Benerator, e.g.: export BENERATOR_HOME=/Developer/Applications/rapiddweller-benerator-ce-3.1.0-jdk-11

3. On Unix/Linux/Mac systems: Set permissions
   Open a shell on the installation's root directory and execute chmod a+x bin/*.sh

4. Mac OS X configuration Set JAVA_HOME On Mac OS X you need to provide benerator with an explicit configuration of the JAVA_HOME path. See http://developer.apple.com/qa/qa2001/qa1170.html for a good introduction to the OS X way of setting up Java. It is based on aliases conventions. If you are not familiar with that, you should read the article. If Java 8 (or newer) is the default version you will use, you can simply define JAVA_HOME by adding the following line to your .profile: in your user directory: export JAVA_HOME=/Library/Java/Home If it does not work or if you need to use different Java versions, it is easier to 'hard-code' JAVA_HOME like this: export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/

Note: We currently recommend following projects for your proper Mac OS X setup:

• brew
• adoptopenjdk
• jenv

Run

If you followed above installation steps carefully, run your benerator scripts from command line.

1. Start benerator from command line

benerator <YOUR_BENERATOR_SCRIPT>.xml

To get started please find some demo scripts in the folder /src/demo/resources/demo of this repository.

Docs / rapiddweller manual

There are various sources to get you started or extend your benerator knowledge:

• Read our docs on our Benerator Manual site or create your own mkdocs with python mkdocs build. Keep in mind that you might need to install certain mkdocs dependencies first.

• Consider the Maven Site Docs or create your own docs using maven mvn site:site. The generated docs include Javadoc, Test Reports and more.

• Download the Benerator Manual from our website.

• Checkout the benerator website www.benerator.de for additional support resources.

If there are still open questions and things are unclear because they are missing or insufficiently explained in our Benerator manual, please open an issue on GitHub, so we can expand our manual to make things clearer for you.

Table of Contents Manual

Navigate to the sources for our generated mkdocs:

• Goals and Features
• Installation
• The Benerator Project Wizard
• Quick tour through the descriptor file format
• Data Generation Concepts
• Data Anonymization Concepts
• Regular Expression Support
• Data File Processing
• XML File Processing
• Distribution Concepts
• Using Relational Databases
• Message Queue Access (Enterprise Edition)
• Kafka Access (Enterprise Edition)
• Advanced Topics
• Generating Unique Data
• Scripting
• rapiddwellerScript
• Command Line Tools
• Domains
• Component Reference
• Using DB Sanity
• Maven Benerator Plugin
• Extending Benerator
• Using Benerator as Load Generator
• NoSQL Support
• Troubleshooting
• Monitoring Benerator
• Benerator Performance Tuning
• Migrating Benerator Projects

Development Quick Start (only Linux and Mac OS X)

If you want to start development or use the maven project to build rapiddweller Benerator by yourself, on Linux or Mac OS X you can also try the quickstart by using the helper scripts. It might be required to run the scripts with sudo.

IMPORTANT: If you want to use the command benerator in your shell session, you have to execute source script/2_setup_benerator.sh
If you want to install Benerator permanently into your system, you have to modify your environment file or your ~/.profile
and add ENV variable BENERATOR_HOME and PATH=$BENERATOR_HOME/bin:$PATH

to execute the scripts you can do ...

bash script/1_install_mvn_dependencies.sh
source script/2_setup_benerator.sh
bash script/3_execute_demos.sh

... alternatively, you can also set execute permissions like ...

chmod +x script/1_install_mvn_dependencies.sh
chmod +x script/2_setup_benerator.sh
chmod +x script/3_execute_demos.sh

... and execute scripts like this

.script/1_install_mvn_dependencies.sh
.script/2_setup_benerator.sh
.script/3_execute_demos.sh

• 1_install_mvn_dependencies.sh : This script is checking Prerequisites for you, cloning all rapiddweller-benerator-ce SubProjects and install via Maven locally.

• 2_setup_benerator.sh : This script is building on script no. 1 and using installed dependencies and packed jar, assemble it to a rapiddweller-benerator-ce.tar.gz and setup Benerator locally into your user home directory.

• 3_execute_demos.sh : This script is building on script no. 2 and use the unpacked and configure rapiddweller-benerator-ce application to execute existing demo files.

Getting Involved

If you would like to reach out to the maintainers, contact us via our Contact-Form or email us at [email protected].

Contributing

Please see our Contributing guidelines. For releasing see our release creation guide. Check out the maintainers website!