JSONLab: compact, portable, robust JSON/binary-JSON encoder/decoder for MATLAB/Octave
Nine years in the making, JSONLab 2.0 (codename: Magnus Prime) has finally arrived!
JSONLab 2.0 is not just a MATLAB/Octave toolbox, but a reference library to the below two new data specifications:
Numerous updates have been added in this milestone release since its previous
version v1.9.8 in Oct. 2019. A list of the major changes are summarized below
(with key features marked by *), including the support to _ArrayShape_ to
efficiently encode special matrices, the addition of jsave/jload to save
and restore variables in MATLAB/Octave like the save/load commands
(experimental), and the associated Python modules (jdata and bjdata):
jdata and bjdata python modules to share data with MATLABPlease note that JSONLab v2.0 is now compliant with JData Spec Draft 3; in comparison, v1.9.8 is compatible with Draft 2; v1.9 and previous releases are compatible with Draft 1. JSONLab v2.0 can read all data files generated by v1.9.8, but v1.9.8 can not read the new UBJSON markers introduced in v2.0.
The newly introduced jsave/jload functions are in the experimental stage.
They generate .jamm files which are renamed binary-JData/UBJSON files;
they can be 50% smaller than .mat files if using jsave(...,'compression','lzma')
and can be readily opened among a long list of programming environments
such as Python, JavaScript and Go.
The saveubjson/loadubjson functions added support to the Binary JData specification (BJData)
v1 Draft-1 (https://github.com/fangq/bjdata) and are now renamed as savebj/loadbj
(saveubjson/loadubjson are kept for compatibility purposes as aliases to the new
functions). The BJData spec is largely compatible with UBJSON spec Draft 12, with the
following differences (we are working with the UBJSON maintainer to merge
these two specifications):
uint16 [u], uint32 [m], uint64 [M] float16 [h] ('''new in JSONLab v2.0''')NaN/Inf/-Inf to null (supported in JSONLab since 2013)To avoid using the new type markers, one should attach 'UBJSON',1 in the savebj
command as
savebj('',data,'FileName','myfile.bjd','UBJSON',1);
To read data files generated by JSONLab v1.9 or older versions, you need to attach
option 'FormatVersion', 1.9 in all the loadjson/savejson function calls.
To convert an older file (JSON/UBJSON) to the new format, you should run
data=loadjson('my_old_data_file.json','FormatVersion',1.9)
savejson('',data,'FileName','new_file.json')
You are strongly encouraged to convert all pre-v1.9.8 generated data files using the new format.
JSONLab is a free and open-source JSON/UBJSON/MessagePack encoder and decoder written in the native MATLAB language. It can be used to convert a MATLAB data structure (array, struct, cell, struct array, cell array, and objects) into JSON/UBJSON/MessagePack formatted strings and files, or to parse a JSON/UBJSON/MessagePack file into MATLAB data structure. JSONLab supports both MATLAB and GNU Octave (a free MATLAB clone).
JSON (JavaScript Object Notation) is a highly portable, human-readable and "fat-free" text format
to represent complex and hierarchical data, widely used for data-exchange in applications.
UBJSON (Universal Binary JSON) is a binary JSON format,
specifically designed to specifically address the limitations of JSON, permitting
efficient storage of binary data with strongly typed data records, resulting in smaller
file sizes and fast encoding and decoding. MessagePack is another binary
JSON-like data format widely used in data exchange in web/native applications.
It is slightly more compact than UBJSON, but is not directly readable compared
to UBJSON.
We envision that both JSON and its binary counterparts will play important rules not only for light-weight data storage, but also for storage and interchange of scientific data. It has both the flexibility and generality as in other general-purpose file specifications, such as HDF5 but has significantly reduced complexity and excellent readability.
Towards this goal, we have developed the JData Specification (http://github.com/fangq/jdata) to standardize serializations of complex scientific data structures, such as N-D arrays, sparse/complex-valued arrays, trees, maps, tables and graphs using JSON/binary JSON constructs. The text and binary formatted JData files are syntactically compatible with JSON/UBJSON formats, and can be readily parsed using existing JSON and UBJSON parsers. JSONLab is not just a parser and writer of JSON/UBJSON data files, but one that systematically converts complex scientific data structures into human-readable and universally supported JSON forms using the standardized JData data annotations.
The installation of JSONLab is no different from installing any other MATLAB toolbox. You only need to download/unzip the JSONLab package to a folder, and add the folder's path to MATLAB/Octave's path list by using the following command:
addpath('/path/to/jsonlab');
If you want to add this path permanently, you need to type pathtool,
browse to the root folder of JSONLab and add to the list, then click "Save".
Then, run rehash in MATLAB, and type "which savejson", if you see an
output, that means JSONLab is installed for MATLAB/Octave.
If you use MATLAB in a shared environment such as a Linux server, the best way to add path is to type
mkdir ~/matlab/
nano ~/matlab/startup.m
and type addpath('/path/to/jsonlab') in this file, save and exit the editor.
MATLAB will execute this file every time it starts. For Octave, the file
you need to edit is ~/.octaverc , where "~" represents your home directory.
To use the data compression features, please download the ZMat toolbox from
https://github.com/fangq/zmat/releases/latest and follow the instruction to
install ZMat first. The ZMat toolbox is required when compression is used on
MATLAB running in the -nojvm mode or GNU Octave, or 'lzma/lzip/lz4/lz4hc'
compression methods are specified. ZMat can also compress large arrays that
MATLAB's Java-based compression API does not support.
JSONLab has been available as an official Fedora package since 2015. You may install it directly using the below command
sudo dnf install octave-jsonlab
To enable data compression/decompression, you need to install octave-zmat using
sudo dnf install octave-zmat
JSONLab is also available on Arch Linux. You may install it using the below command
sudo pikaur -S jsonlab
jsave/jload to share workspaceStarting from JSONLab v2.0, we provide a pair of functions, jsave/jload to store
and retrieve variables from the current workspace, similar to the save/load
functions in MATLAB and Octave. The files that jsave/jload reads/writes is by
default a binary JData file with a suffix .jamm. The file size is comparable
(can be smaller if use lzma compression) to .mat files. This feature
is currently experimental.
The main benefits of using .jamm file to share matlab variables include
.jamm file can be 50% smaller than a .mat file when using jsave(..., "compression","lzma"); the only drawback is longer saving time..jamm file can be readily read/opened among many programming environments, including .mat file support is not generally available. .jamm is largely compatible with UBJSON's parsers available at .jamm file is quasi-human-readable, one can see the internal data fields strings -n 2 file.jamm | astyle, jsave/jload can also use MessagePack and JSON formats as the underlying jsave % save the current workspace to jamdata.jamm
jsave mydata.jamm
jsave('mydata.jamm','vars',{'var1','var2'})
jsave('mydata.jamm','compression','lzma')
jsave('mydata.json','compression','gzip')
jload % load variables from jamdata.jamm to the current workspace
jload mydata.jamm % load variables from mydata.jamm
vars=jload('mydata.jamm','vars',{'var1','var2'}) % return vars.var1, vars.var2
jload('mydata.jamm','simplifycell',0)
jload('mydata.json')
Despite the use of portable data annotation defined by the JData Specification,
the output JSON files created by JSONLab are 100% JSON compatible (with
the exception that long strings may be broken into multiple lines for better
readability). Therefore, JSONLab-created JSON files (.json, .jnii, .jnirs etc)
can be readily read and written by nearly all existing JSON parsers, including
the built-in json module parser in Python.
However, we strongly recommend one to use a lightweight jdata module,
developed by the same author, to perform the extra JData encoding and decoding
and convert JSON data directly to convenient Python/Numpy data structures.
The jdata module can also directly read/write UBJSON/Binary JData outputs
from JSONLab (.bjd, .ubj, .bnii, .bnirs, .jamm etc). Using binary JData
files are exptected to produce much smaller file sizes and faster parsing,
while maintainining excellent portability and generality.
In short, to conveniently read/write data files created by JSONLab into Python, whether they are JSON based or binary JData/UBJSON based, one just need to download the below two light-weight python modules:
To install these modules on Python 2.x, please first check if your system has
pip and numpy, if not, please install it by running (using Ubuntu/Debian as example)
sudo apt-get install python-pip python3-pip python-numpy python3-numpy
After the installation is done, one can then install the jdata and bjdata modules by
pip install jdata --user
pip install bjdata --user
To install these modules for Python 3.x, please replace pip by pip3.
If one prefers to install these modules globally for all users, simply
execute the above commands using sudo and remove the --user flag.
The above modules require built-in Python modules json and NumPy (numpy).
Once the necessary modules are installed, one can type python (or python3), and run
import jdata as jd
import numpy as np
from collections import OrderedDict
data1=jd.loadt('myfile.json',object_pairs_hook=OrderedDict);
data2=jd.loadb('myfile.ubj',object_pairs_hook=OrderedDict);
data3=jd.loadb('myfile.jamm',object_pairs_hook=OrderedDict);
where jd.loadt() function loads a text-based JSON file, performs
JData decoding and converts the enclosed data into Python dict, list
and numpy objects. Similarly, jd.loadb() function loads a binary
JData/UBJSON file and performs similar conversions. One can directly call
jd.load() to open JSONLab (and derived toolboxes such as jnifti:
https://github.com/fangq/jnifti or jsnirfy: https://github.com/fangq/jsnirfy)
generated files based on their respective file suffix.
Similarly, the jd.savet(), jd.saveb() and jd.save functions
can revert the direction and convert a Python/Numpy object into JData encoded
data structure and store as text-, binary- and suffix-determined output files,
respectively.
JSONLab v1.9.8 is the beta release of the next milestone - code named "Magnus".
Starting from this release, JSONLab supports encoding/decoding MessagePack, a widely-used binary JSON-like data format. Via ZMat v0.9, JSONLab v1.9.8 also supports LZMA/LZ4/LZ4HC data compression/decompression. More importantly, JSONLab is now the official reference implementation for JData Specification (Draft 2) as defined in http://github.com/fangq/jdata, the foundation for the OpenJData Project (http://openjdata.org).
There have been numerous major updates to this toolbox since the previous release v1.9 in May 2019. A list of the major changes are summarized below with key features marked by *:
Please note that JSONLab v1.9.8 is compliant with JData Spec Draft 2, while v1.9 and previous releases are compatible with Draft 1. The main differences are
_ArrayCompressionMethod_, _ArrayCompressionSize_ and _ArrayCompressedData_
were replaced by _ArrayZipType_, _ArrayZipSize_ and _ArrayZipData_, respectively_ArrayData_ was changed from column-major to
row-majorTo read data files generated by JSONLab v1.9 or older versions, you need to attach
option 'FormatVersion', 1.9 in all loadjson/savejson function calls.
To convert an older file (JSON/UBJSON) to the new format, you should run
data=loadjson('my_old_data_file.json','FormatVersion',1.9)
savejson('',data,'FileName','new_file.json')
You are strongly encouraged to convert all previously generated data files using the new format.
JSONLab is a free and open-source implementation of a JSON/UBJSON/MessagePack encoder and a decoder in the native MATLAB language. It can be used to convert a MATLAB data structure (array, struct, cell, struct array, cell array, and objects) into JSON/UBJSON/MessagePack formatted strings, or to decode a JSON/UBJSON/MessagePack file into MATLAB data structure. JSONLab supports both MATLAB and GNU Octave (a free MATLAB clone).
JSON (JavaScript Object Notation, http://json.org/ ) is a highly portable, human-readable and "fat-free" text format to represent complex and hierarchical data. It is as powerful as XML, but less verbose. JSON format is widely used for data-exchange in applications.
UBJSON (Universal Binary JSON, http://ubjson.org/) is a binary JSON format,
specifically optimized for compact file size and better performance while keeping
the semantics as simple as the text-based JSON format. Using the UBJSON
format allows to wrap complex binary data in a flexible and extensible
structure, making it possible to process complex and large dataset without accuracy
loss due to text conversions. MessagePack is another binary JSON-like data
format widely used in data exchange in web/native applications. It is slightly more
compact than UBJSON, but is not directly readable compared to UBJSON.
We envision that both JSON and its binary counterparts will play important roles as mainstream data-exchange formats for scientific research. It has both the flexibility and generality as offered by other popular general-purpose file specifications, such as HDF5 but with significantly reduced complexity and excellent readability.
Towards this goal, we have developed the JData Specification (http://github.com/fangq/jdata) to standardize serializations of complex scientific data structures, such as N-D arrays, sparse/complex-valued arrays, trees, maps, tables and graphs using JSON/binary JSON constructs. The text and binary formatted JData files are syntactically compatible with JSON/UBJSON formats, and can be readily parsed using existing JSON and UBJSON parsers.
Please note that data files produced by saveubjson may utilize a special
"optimized header" to store N-D (N>=2) arrays, as defined in the JData Specification Draft 2.
This feature is not supported by UBJSON Specification Draft 12. To produce
UBJSON files that can be parsed by UBJSON-Draft-12 compliant parsers, you must
add the option 'NestArray',1 in the call to saveubjson.
The installation of JSONLab is no different from installing any other MATLAB toolbox. You only need to download/unzip the JSONLab package to a folder, and add the folder's path to MATLAB/Octave's path list by using the following command:
addpath('/path/to/jsonlab');
If you want to add this path permanently, you can type "pathtool",
browse to the JSONLab root folder and add to the list, then click "Save".
Then, run "rehash" in MATLAB, and type "which savejson", if you see an
output, that means JSONLab is installed for MATLAB/Octave.
If you use MATLAB in a shared environment such as a Linux server, the best way to add path is to type
mkdir ~/matlab/
nano ~/matlab/startup.m
and type addpath('/path/to/jsonlab') in this file, save and quit the editor.
MATLAB will execute this file every time it starts. For Octave, the file
you need to edit is ~/.octaverc , where "~" is your home directory.
JSONLab has been available as an official Fedora package since 2015. You may install it directly using the below command
sudo dnf install octave-jsonlab
To enable data compression/decompression, you are encouraged to install octave-zmat using
sudo dnf install octave-zmat
JSONLab is also available on Arch Linux. You may install it using the below command
sudo pacman -S jsonlab
JSONlab ChangeLog (key features marked by *):
== JSONlab 1.9 (codename: Magnus - alpha), FangQ <q.fang
2019-05-06 [25ad795] unescape strings in loadjson.m 2019-05-04 [2e317c9] explain extra compression fields 2019-05-02 [1b1be65] avoid side effect of removing singletarray 2019-05-02*[8360fd1] support zmat based base64 encoding and decoding 2019-05-01*[c797bb2] integrating zmat, for zlib/gzip data compression 2019-04-29 [70551fe] remove warnings from matlab 2019-04-28 [0d61c4b] complete data compression support, close #52 2019-04-27 [804115b] avoid typecast error 2019-04-27 [c166aa7] change default compressarraysize to 100 2019-04-27*[3322f6f] major new feature: support array compression and decompression 2019-03-13*[9c01046] support saving function handles, close #51 2019-03-13 [a8fde38] add option to parse string array or convert to char, close #50 2019-03-12 [ed2645e] treat string array as cell array in newer matlab 2018-11-18 [c3eb021] allow saving uint64 integers in saveubjson, fix #49
JSONlab ChangeLog (key features marked by *):
== JSONlab 1.8 (codename: Nominus - final), FangQ <q.fang (at) neu.edu> ==
2018-07-12 [03a6c25] update documentation, bump version to 1.8, tag Nominus 2018-07-12 [1597106] add patch provided by pjkoprowski to support MATLAB table, fix #29 2018-07-12 [f16cc57] fix #31, throw an error when : array construct is used 2018-07-12 [956e000] drop octave 3.x support, fix ubjson error in octave 2018-07-12 [e090f0a] fix octave warning for saveubjson 2018-07-12 [34284c7] fix issues #34 #39 #44 and #45, support double-quited strings 2017-09-06 [474d8c8] Merge pull request #41 from dasantonym/master 2017-08-07 [38b24fb] added package.json to be able to intall via npm, converted readme to utf-8, added .gitignore 2017-07-19 [ae7a5d9] Merge pull request #40 from astorfi/master 2017-07-17 [3176d44] Add files via upload 2017-07-17 [154ef61] Rename README.txt to README.rst 2017-03-27 [31b5bdc] simplify condition flow in matching_bracket 2017-03-27 [86ef12a] avoid error in matlab 2017a, close #34 2017-02-18 [4a09ac3] Merge pull request #32 from vrichter/master 2017-02-14 [e67d3a3] respect integer types
JSONlab ChangeLog (key features marked by *):
== JSONlab 1.5 (codename: Nominus - alpha), FangQ <q.fang (at) neu.edu> ==
2017/01/02 *use Big-endian format to store floating points (d/D) in saveubjson (Issue #25) 2017/01/02 *speedup parsing large unstructured data by 2x (Issue #9) 2017/01/01 make parsing independent of white space (Issue #30) 2016/08/27 allow to parse array of homogeneous elements (Issue #5) 2016/08/22 permit [] inside file names in savejson 2016/01/06 fix a bug that prevents saving to a file in savejson
JSONlab ChangeLog (key features marked by *):
== JSONlab 1.2 (codename: Optimus - Update 2), FangQ <q.fang (at) neu.edu> ==
2015/12/16 replacing string concatenation by str cells to gain 2x speed in savejson (Issue #17) 2015/12/11 fix FileName option case bug (SVN rev#495) 2015/12/11 add SingletCell option, add SingletArray to replace NoRowBracket (Issue #15,#8) 2015/11/10 fix bug for inerpreting file names as JSON string - by Mykhailo Bratukha (Pull #14) 2015/10/16 fix bug for cell with transposed data - by Insik Kim (Pull #12) 2015/09/25 support exporting matlab object to JSON - by Sertan Senturk (Pull #10, #11)