omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JSON, and custom formats) in streaming fashion and transforms data into desired JSON output based on a schema written in JSON.
Min Golang Version: 1.14
Omniparser is publicly available under MIT License. Individual and corporate sponsorships are welcome and gratefully appreciated, and will be listed in the SPONSORS page. Company-level sponsors get additional benefits and supports granted in the COMPANY LICENSE.
Docs:
transform_declarations
.custom_func
, Specially javascript
: An in depth look of how custom_func
is used, specially the all mighty javascript
(and javascript_with_context
).References:
Examples:
In the example folders above you will find pairs of input files and their schema files. Then in the
.snapshots
sub directory, you'll find their corresponding output files.
Use The Playground (may need to wait for a few seconds for instance to wake up)
for trying out schemas and inputs, yours or existing samples, to see how ingestion and transform work.
As for now (2023/03/14), all of our previous free docker hosting solutions went away and we haven't found another one yet. For now please clone the repo and use ./cli.sh
as described in the Getting Started page.
csv2
file format that supersedes the original csv
format with support of hierarchical and nested records.fixedlength2
file format that supersedes the original fixed-length
format with support of hierarchical and nested envelopes.Transform.RawRecord()
for caller of omniparser to access the raw ingested record.custom_parse
in favor of custom_func
(custom_parse
is still usable for
back-compatibility, it is just removed from all public docs and samples).NonValidatingReader
EDI segment reader.omni.2.1
due a number of incompatible schema changes:
'result_type'
-> 'type'
'ignore_error_and_return_empty_str
-> 'ignore_error'
'keep_leading_trailing_space'
-> 'no_trim'
times.SmartParse
.transform
cmd or long-running http server
mode).javascript
engine integration as a custom_func.