Compress json-data based on its json-schema while still having valid json
Compress json-data based on its json-schema while still having valid json. It works by compressing long attribute-names into smaller ones and backwards.
For example this:
{
"firstName": "Corrine",
"lastName": "Ziemann",
"title": "Ms.",
"gender": "f",
"zipCode": 75963,
"countryCode": "en",
"birthYear": 1960,
"active": false,
"shoppingCartItems": [
{
"productNumber": 29857,
"amount": 1
},
{
"productNumber": 53409,
"amount": 6
}
]
}
becomes this:
{
"|e": "Corrine",
"|g": "Ziemann",
"|j": "Ms.",
"|f": "f",
"|k": 75963,
"|d": "en",
"|c": 1960,
"|a": false,
"|i": [
{
"|h": 29857,
"|b": 1
},
{
"|h": 53409,
"|b": 6
}
]
}
The efficiency depends on the amount and length of the attribute names.
You can reproduce these results by running npm run test:efficiency
.
The compression works pretty fast. Here are some time measurements on a single intel i7 CPU.
0.02ms
0.021ms
0.027ms
per objectYou can reproduce these results by running npm run test:performance
.
Gzip generates its compression-flags from the input. This makes it more efficient, the more data is compressed at once. But gzip is less efficient the smaller the dataset is. The key-compression creates the compression-table from the jsonschema up front with has advantages when small pieces of data are compressed.
npm install jsonschema-key-compression --save
Creates a compression-table from the json-schema.
import {
createCompressionTable
} from 'jsonschema-key-compression';
const compressionTable = createCompressionTable(jsonSchema);
Compress a json-object based on its schema.
import {
compressObject
} from 'jsonschema-key-compression';
const compressedObject = compressObject(
compressionTable,
jsonObject
);
Decompress a compressed object.
import {
decompressObject
} from 'jsonschema-key-compression';
const jsonObject = decompressObject(
compressionTable,
compressedObject
);
Transform a chain of json-attributes into its compressed format.
import {
compressedPath
} from 'jsonschema-key-compression';
const compressed = compressedPath(
compressionTable,
'whateverNested.firstName'
); // > '|a.|b'
Decompress a compressed path.
import {
decompressedPath
} from 'jsonschema-key-compression';
const decompressed = decompressedPath(
compressionTable,
'|a.|b' // from compressedPath
); // > 'whateverNested.firstName'
Compress a mango-query so that it can run over a NoSQL-database that has stored compressed documents.
import {
compressQuery
} from 'jsonschema-key-compression';
const compressed = compressQuery(
compressionTable,
{
selector: {
active: {
$eq: true
}
},
skip: 1,
limit: 1,
fields: [
'id',
'name'
],
sort: [
'name'
]
}
);
Transforms a json-schema into a compressed form, so that it can be used to validate compressed objects.
import {
createCompressedJsonSchema
} from 'jsonschema-key-compression';
const schema = {
type: 'object',
properties: {
firstName: {
type: 'string'
}
},
required: [
'firstName'
]
}
const compressedSchema = createCompressedJsonSchema(
compressionTable,
schema
);
console.dir(compressedSchema);
/**
{
type: 'object',
properties: {
|a: {
type: 'string'
}
},
required: [
'|a'
]
}
*/
This module was originally created for the RxDB compression plugin but in theory it can be used in any json based state management system or database.