A low-level, lightweight protocol buffers implementation in JavaScript.
A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding protocol buffers, a compact binary format for structured data serialization. Works both in Node and the browser. Supports lazy decoding and detailed customization of the reading/writing code.
This library is extremely fast — much faster than native JSON.parse
/JSON.stringify
and the protocol-buffers module.
Here's a result from running a real-world benchmark on Node v6.5
(decoding and encoding a sample of 439 vector tiles, 22.6 MB total):
Install pbf
and compile a JavaScript module from a .proto
file:
$ npm install -g pbf
$ pbf example.proto > example.js
Then read and write objects using the module like this:
var Pbf = require('pbf');
var Example = require('./example.js').Example;
// read
var pbf = new Pbf(buffer);
var obj = Example.read(pbf);
// write
var pbf = new Pbf();
Example.write(obj, pbf);
var buffer = pbf.finish();
Alternatively, you can compile a module directly in the code:
var compile = require('pbf/compile');
var schema = require('protocol-buffers-schema');
var proto = schema.parse(fs.readFileSync('example.proto'));
var Test = compile(proto).Test;
If you use webpack
as your module bundler, you can use pbf-loader
to load .proto files directly. It returns a compiled module ready to be used.
Given you already configured your webpack.config.js
, the code above would look like:
var Pbf = require('pbf');
var proto = require('./example.proto');
var Test = proto.Test;
var data = new Pbf(buffer).readFields(readData, {});
function readData(tag, data, pbf) {
if (tag === 1) data.name = pbf.readString();
else if (tag === 2) data.version = pbf.readVarint();
else if (tag === 3) data.layer = pbf.readMessage(readLayer, {});
}
function readLayer(tag, layer, pbf) {
if (tag === 1) layer.name = pbf.readString();
else if (tag === 3) layer.size = pbf.readVarint();
}
var pbf = new Pbf();
writeData(data, pbf);
var buffer = pbf.finish();
function writeData(data, pbf) {
pbf.writeStringField(1, data.name);
pbf.writeVarintField(2, data.version);
pbf.writeMessage(3, writeLayer, data.layer);
}
function writeLayer(layer, pbf) {
pbf.writeStringField(1, layer.name);
pbf.writeVarintField(2, layer.size);
}
Node and Browserify:
npm install pbf
Making a browser build:
npm install
npm run build-dev # dist/pbf-dev.js (development build)
npm run build-min # dist/pbf.js (minified production build)
CDN link: https://unpkg.com/[email protected]/dist/pbf.js
Create a Pbf
object, optionally given a Buffer
or Uint8Array
as input data:
// parse a pbf file from disk in Node
var pbf = new Pbf(fs.readFileSync('data.pbf'));
// parse a pbf file in a browser after an ajax request with responseType="arraybuffer"
var pbf = new Pbf(new Uint8Array(xhr.response));
Pbf
object properties:
pbf.length; // length of the underlying buffer
pbf.pos; // current offset for reading or writing
Read a sequence of fields:
pbf.readFields(function (tag) {
if (tag === 1) pbf.readVarint();
else if (tag === 2) pbf.readString();
else ...
});
It optionally accepts an object that will be passed to the reading function for easier construction of decoded data,
and also passes the Pbf
object as a third argument:
var result = pbf.readFields(callback, {})
function callback(tag, result, pbf) {
if (tag === 1) result.id = pbf.readVarint();
}
To read an embedded message, use pbf.readMessage(fn[, obj])
(in the same way as read
).
Read values:
var value = pbf.readVarint();
var str = pbf.readString();
var numbers = pbf.readPackedVarint();
For lazy or partial decoding, simply save the position instead of reading a value, then later set it back to the saved value and read:
var fooPos = -1;
pbf.readFields(function (tag) {
if (tag === 1) fooPos = pbf.pos;
});
...
pbf.pos = fooPos;
pbf.readMessage(readFoo);
Scalar reading methods:
readVarint(isSigned)
(pass true
if you expect negative varints)readSVarint()
readFixed32()
readFixed64()
readSFixed32()
readSFixed64()
readBoolean()
readFloat()
readDouble()
readString()
readBytes()
skip(value)
Packed reading methods:
readPackedVarint(arr, isSigned)
(appends read items to arr
)readPackedSVarint(arr)
readPackedFixed32(arr)
readPackedFixed64(arr)
readPackedSFixed32(arr)
readPackedSFixed64(arr)
readPackedBoolean(arr)
readPackedFloat(arr)
readPackedDouble(arr)
Write values:
pbf.writeVarint(123);
pbf.writeString("Hello world");
Write an embedded message:
pbf.writeMessage(1, writeObj, obj);
function writeObj(obj, pbf) {
pbf.writeStringField(obj.name);
pbf.writeVarintField(obj.version);
}
Field writing methods:
writeVarintField(tag, val)
writeSVarintField(tag, val)
writeFixed32Field(tag, val)
writeFixed64Field(tag, val)
writeSFixed32Field(tag, val)
writeSFixed64Field(tag, val)
writeBooleanField(tag, val)
writeFloatField(tag, val)
writeDoubleField(tag, val)
writeStringField(tag, val)
writeBytesField(tag, buffer)
Packed field writing methods:
writePackedVarint(tag, val)
writePackedSVarint(tag, val)
writePackedSFixed32(tag, val)
writePackedSFixed64(tag, val)
writePackedBoolean(tag, val)
writePackedFloat(tag, val)
writePackedDouble(tag, val)
Scalar writing methods:
writeVarint(val)
writeSVarint(val)
writeSFixed32(val)
writeSFixed64(val)
writeBoolean(val)
writeFloat(val)
writeDouble(val)
writeString(val)
writeBytes(buffer)
Message writing methods:
writeMessage(tag, fn[, obj])
writeRawMessage(fn[, obj])
Misc methods:
realloc(minBytes)
- pad the underlying buffer size to accommodate the given number of bytes;
note that the size increases exponentially, so it won't necessarily equal the size of data writtenfinish()
- make the current buffer ready for reading and return the data as a buffer slicedestroy()
- dispose the bufferFor an example of a real-world usage of the library, see vector-tile-js.
If installed globally, pbf
provides a binary that compiles proto
files into JavaScript modules. Usage:
$ pbf <proto_path> [--no-write] [--no-read] [--browser]
The --no-write
and --no-read
switches remove corresponding code in the output.
The --browser
switch makes the module work in browsers instead of Node.
The resulting module exports each message by name with the following methods:
read(pbf)
- decodes an object from the given Pbf
instancewrite(obj, pbf)
- encodes an object into the given Pbf
instance (usually empty)The resulting code is clean and simple, so feel free to customize it.
This release include tons of compatibility/robustness fixes, and a more reliable Node implementation. Decoding performance is expected to get up to ~15% slower than v2.0 in Node (browsers are unaffected), but encoding got faster by ~15% in return.
Uint8Array
instead of Buffer
internally (and produce corresponding result on finish()
), making it fully match the browser implementation for consistency and simplicity.writeVarint
to write 0
when given NaN
or other non-number to avoid producing a broken Protobuf message.readPacked*
methods signature to accept an optional arr
argument to append the results to (to support messages with repeated fields that mix packed/non-packed encoding).isSigned
argument to readVarint
that enables proper reading of negative varints.readVarint64()
(it still works, but it's recommended to be changed to readVarint(true)
).[]
if they're not present.packed=false
being interpreted as packed.writeVarint
that affected certain numbers.writeVarint
that affected certain numbers.syntax
keyword proto files (by updating resolve-protobuf-schema
dependency).writeRawMessage
method for writing a message without a tag, useful for creating pbfs with multiple top-level messages.readVarint64
method for proper decoding of negative int64
-encoded values.pbf
binary that compiles .proto
files into Pbf
-based JavaScript modules.writeMessage
signature to (tag, fn, obj)
(see example in the docs)
for a huge encoding performance improvement.readPacked
and writePacked
methods that accept type as a string
with readPackedVarint
, etc. for each type (better performance and simpler API).readPacked
and writePacked
performance (the tile reading benchmark is now 70% faster).Brings tons of improvements and fixes over the previous version (0.0.2
).
Basically makes the library complete.
readFields
and readMessage
methods for a much easier reading API.readFloat
, readBoolean
, readSFixed32
, readSFixed64
.writeUInt64
, writeSFixed32
, writeSFixed64
.readDouble
and readString
to use native Buffer methods under Node.readString
and writeString
to use HTML5 TextEncoder
and TextDecoder
where available.Pbf
buffer
argument optional.readBuffer
/writeBuffer
to readBytes
/writeBytes
.readUInt32
/writeUInt32
to readFixed32
/writeFixed32
, etc.writeTaggedVarint
to writeVarintField
, etc.writePacked
signature from (type, tag, items)
to (tag, type, items)
.readVarint
to handle varints bigger than 6 bytes.readSVarint
to handle number bigger than 2^30
.writeVarint
failing on some integers.writeVarint
not throwing an error on numbers that are too big.readUInt64
always failing.