Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable
This release adds new functionality to the ETL utilities and also adds Windows supports for those utilities.
A new ETL utility supports the execution of an existing ETL job from the shell or command line based on the ETL name. This is helpful during the diagnostic processing of ETLs where it's preferable to (re)run an ETL.
The utility kit now supports running ETL Log Utilities on the Windows command line. Windows users can now use the command line to show all log files associated with an ETL, show the logs of ETLs that were executed today, summarize the run-times for ETLs associated with a particular datasets, and running a specific ETL based on the name of the dataset. By adding Windows compatibility, Windows users now have full use of the Open Data ETL Utility Kit.
This release also includes some minor corrections to the documentation.
This release adds DataSync within the library to ensure compatibility between the ETL framework and DataSync, which is a dependency. This release of the framework is compatible with DataSync 1.5.3. While it may be compatible with newer versions of DataSync, that has not always been the case without modification to the framework as DataSync have evolved. In addition, other changes were made to the framework to maintain DataSync compatibility.
This is the initial public release of the Open Data ETL Utility Kit. This repository helps alleviate the need to engineer an ETL process to automatically populate an open data portal. Documentation located at http://open-data-etl-utility-kit.readthedocs.org/en/latest/