CBOR codec (RFC 8949) with CBOR tags, Go struct tags (toarray, keyasint, omitempty), float64/32/16, big.Int, and fuzz tested billions of execs.
This release adds important new features, optimizations, bug fixes, and 3 new contributors. Fuzz tests passed 5+ billion execs for CBOR encoding and decoding functions. Minimum officially supported Go is bumped to Go 1.17.
interface{}
as *big.Int
by @fxamacker in https://github.com/fxamacker/cbor/pull/456
interface{}
by @fxamacker in https://github.com/fxamacker/cbor/pull/457
any
by @ssuriyan7 in https://github.com/fxamacker/cbor/pull/475
Improve memory allocs 🗜️ and speed 🚀 of encoding maps by using Go 1.18-1.20 features by @dinhxuanvu in https://github.com/fxamacker/cbor/pull/468
benchstat
results provided by @dinhxuanvu
NOTE: Go 1.17 is the minimum version supported by v2.6.0. Go 1.20+ will use optimizations introduced by PR 468.
UnrecognizedTagToAny
option to TestDecOptions
by @fxamacker in https://github.com/fxamacker/cbor/pull/484
*cbor.SimpleValue
by @fxamacker in https://github.com/fxamacker/cbor/pull/461
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0...v2.6.0
interface{}
as *big.Int
by @fxamacker in https://github.com/fxamacker/cbor/pull/456
interface{}
by @fxamacker in https://github.com/fxamacker/cbor/pull/457
any
by @ssuriyan7 in https://github.com/fxamacker/cbor/pull/475
Improve memory allocs 🗜️ and speed 🚀 of encoding maps by using Go 1.18-1.20 features by @dinhxuanvu in https://github.com/fxamacker/cbor/pull/468
benchstat
results provided by @dinhxuanvu
NOTE: Go 1.17 is the minimum version supported by v2.6.0. Go 1.20+ will use optimizations introduced by PR 468.
UnrecognizedTagToAny
option to TestDecOptions
by @fxamacker in https://github.com/fxamacker/cbor/pull/484
*cbor.SimpleValue
by @fxamacker in https://github.com/fxamacker/cbor/pull/461
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0...v2.6.0-rc.0
This release adds important new features, optimizations, bug fixes, and 8 new contributors. Fuzz tests passed 5+ billion execs for CBOR encoding and decoding functions.
Notable changes are split into 3 categories.
These new features and bug fixes were cherry-picked to highlight for review (for projects using older version).
SimpleValue
type to more fully support CBOR Simple Values, including values not assigned by IANA and...ByteString
type to support CBOR maps with byte string keys because Go doesn't allow []byte as map keys and...Decoder.Decode()
return io.ErrUnexpectedEOF
instead of io.EOF
on EOF if current CBOR data item is incomplete.Unmarshal()
and Valid()
return cbor.ExtraneousDataError
(instead of ignoring extraneous data if any remain).DecOptions.UTF8
to decode invalid UTF-8. Default is unchanged (reject invalid UTF-8 and return error).MaxNestedLevels
setting to be configured up to 65535.EncOptions.NilContainersMode
to encode nil Go maps and slices as either CBOR nil (default) or empty container.Decoder.Skip()
to skip CBOR data item in CBOR Sequences (RFC 8742).UnmarshalFirst()
to decode CBOR data item and also return remaining bytes.Valid
and add Wellformed
to replace it.UnmarshalFirst()
to DecMode
interface.Decoder.Buffered
to return remaining data in buffer.RawMessage
has sufficient capacity.Encode()
. It adds a bit of overhead to Encode()
but NewEncoder().Encode()
is faster and uses less memory.Benchmarks provided by @x448 for NewEncoder().Encode()
optimized in PR 382.
$ benchstat bench-v2.4.0.log bench-f9e6291.log
goos: linux
goarch: amd64
pkg: github.com/fxamacker/cbor/v2
cpu: 12th Gen Intel(R) Core(TM) i7-12700H
│ bench-v2.4.0.log │ bench-f9e6291.log │
│ sec/op │ sec/op vs base │
NewEncoderEncode/Go_bool_to_CBOR_bool-20 236.70n ± 2% 58.04n ± 1% -75.48% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20 238.00n ± 2% 63.93n ± 1% -73.14% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20 238.65n ± 2% 64.88n ± 1% -72.81% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20 242.00n ± 2% 63.00n ± 1% -73.97% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20 245.60n ± 1% 68.55n ± 1% -72.09% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20 243.20n ± 3% 68.39n ± 1% -71.88% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20 563.0n ± 2% 378.3n ± 0% -32.81% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20 2.043µ ± 2% 1.906µ ± 2% -6.75% (p=0.000 n=10)
geomean 349.7n 122.7n -64.92%
│ bench-v2.4.0.log │ bench-f9e6291.log │
│ B/op │ B/op vs base │
NewEncoderEncode/Go_bool_to_CBOR_bool-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20 544.0 ± 0% 416.0 ± 0% -23.53% (p=0.000 n=10)
geomean 153.4 ? ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean
│ bench-v2.4.0.log │ bench-f9e6291.log │
│ allocs/op │ allocs/op vs base │
NewEncoderEncode/Go_bool_to_CBOR_bool-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20 28.00 ± 0% 26.00 ± 0% -7.14% (p=0.000 n=10)
geomean 2.782 ? ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean
All merged pull requests are listed here (including the already highlighted ones).
Most coding changes here were already mentioned.
Decoder.Buffered
to return buffered data by @fxamacker in https://github.com/fxamacker/cbor/pull/412
Diagnose()
to return io.EOF
error on empty data by @fxamacker in https://github.com/fxamacker/cbor/pull/410
Non-coding contibutions were made by opening notable issues that directly or indirectly improved this release.
Encode()
returning buffer to pool as default behavior.Full Changelog: https://github.com/fxamacker/cbor/compare/v2.4.0...v2.5.0
v2.5.0-beta5 is fuzz tested and production quality, but docs need to be updated before releasing v2.5.0. IMPORTANT: Please view these release notes (esp. v2.5.0-beta) before upgrading from v2.4 or earlier:
Diagnose
to return io.EOF
on empty data.Diagnose
, DiagnoseFirst
, UnmarshalFirst
, Wellformed
.Decoder
if io.Reader
's Read()
returns 0 bytes read with nil error.Decoder.Buffered
to return remaining data in buffer by @fxamacker in https://github.com/fxamacker/cbor/pull/412
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta4...v2.5.0-beta5
Features already present in v2.4 are release quality. Newly added functions require more fuzzing and docs.
IMPORTANT: Please view these release notes (esp. v2.5.0-beta) before upgrading from v2.4:
Diagnose()
, DiagnoseFirst()
, UnmarshalFirst()
, Wellformed()
.Diagnose()
to return io.EOF
error on empty data by @fxamacker in https://github.com/fxamacker/cbor/pull/410
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta3...v2.5.0-beta4
This release adds 4 new functions: UnmarshalFirst
, Diagnose
, DiagnoseFirst
, and Wellformed
. In addition Valid
function is deprecated (use Wellformed
instead).
UnmarshalFirst
decodes first CBOR data item and returns remaining bytes. This is useful for CBOR Sequences (RFC 8742) and also because Unmarshal
was fixed in v2.5.0-beta to return ExtraneousDataError
if there are remaining bytes.Diagnose
and DiagnoseFirst
returns human-readable Extended Diagnostic Notation (RFC 8610 Appendix G) for the given CBOR encoding. These are useful for debugging and allows more user-friendly error logging.Thank you @immesys, @zensh, and @x448 for your contributions!
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta2...v2.5.0-beta3
There are fewer changes in v2.5.0-beta2 compared to v2.5.0-beta. Speed and memory improvements since v2.4.0 are from v2.5.0-beta. Tests cover 98.5% and reached 100% coverage for stream.go (e.g. Decoder using the provided io.Reader).
Go language recommends that Read
implementations for io.Reader avoid returning 0 with nil error. It also recommends that callers receiving 0 and nil from Read
should treat it as if nothing happened. This edge case is handled properly now.
@x448 shared benchmarks comparison of v2.4.0 vs v2.5.0-beta2. Performance improvements are from v2.5.0-beta.
$ benchstat benchNewEncoderEncode-v2.4.0.log benchNewEncoderEncode-f9e6291.log
goos: linux
goarch: amd64
pkg: github.com/fxamacker/cbor/v2
cpu: 12th Gen Intel(R) Core(TM) i7-12700H
│ benchNewEncoderEncode-v2.4.0.log │ benchNewEncoderEncode-f9e6291.log │
│ sec/op │ sec/op vs base │
NewEncoderEncode/Go_bool_to_CBOR_bool-20 236.70n ± 2% 58.04n ± 1% -75.48% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20 238.00n ± 2% 63.93n ± 1% -73.14% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20 238.65n ± 2% 64.88n ± 1% -72.81% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20 242.00n ± 2% 63.00n ± 1% -73.97% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20 245.60n ± 1% 68.55n ± 1% -72.09% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20 243.20n ± 3% 68.39n ± 1% -71.88% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20 563.0n ± 2% 378.3n ± 0% -32.81% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20 2.043µ ± 2% 1.906µ ± 2% -6.75% (p=0.000 n=10)
geomean 349.7n 122.7n -64.92%
│ benchNewEncoderEncode-v2.4.0.log │ benchNewEncoderEncode-f9e6291.log │
│ B/op │ B/op vs base │
NewEncoderEncode/Go_bool_to_CBOR_bool-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20 128.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20 544.0 ± 0% 416.0 ± 0% -23.53% (p=0.000 n=10)
geomean 153.4 ? ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean
│ benchNewEncoderEncode-v2.4.0.log │ benchNewEncoderEncode-f9e6291.log │
│ allocs/op │ allocs/op vs base │
NewEncoderEncode/Go_bool_to_CBOR_bool-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20 2.000 ± 0% 0.000 ± 0% -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20 28.00 ± 0% 26.00 ± 0% -7.14% (p=0.000 n=10)
geomean 2.782 ? ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta...v2.5.0-beta2
This release is larger than usual because it has been a year since last release. It adds new features, bug fixes, and 8 new contributors.
All unit tests pass (98.4% coverage). Fuzz tests passed hundreds of millions of execs. Fuzzing will continue until v2.5.0 is tagged.
These new features and bug fixes were cherry-picked to highlight for review (for projects using older version in production).
PR 370: Add SimpleValue
type to more fully support CBOR Simple Values, including values not assigned by IANA and...
PR 376: Add ByteString
type to support CBOR maps with byte string keys because Go doesn't allow []byte as map keys and...
PR 379: Make Decoder.Decode()
return io.ErrUnexpectedEOF
instead of io.EOF
on EOF if current CBOR data item is incomplete.
PR 380: Make Unmarshal()
and Valid()
return cbor.ExtraneousDataError
(instead of ignoring extraneous data if any remain).
PR 335: Reuse underlying array if RawMessage
has sufficient capacity.
PR 382: Return buffer to pool in Encode()
. It adds a bit of overhead to Encode()
but NewEncoder().Encode()
is a lot faster and uses less memory.
PR 342: Add DecOptions.UTF8
to decode invalid UTF-8. Default is unchanged (reject invalid UTF-8 and return error).
PR 355 Allow MaxNestedLevels setting to be configured up to 65535.
PR 352, 377: Add EncOptions.NilContainersMode
to encode nil Go maps and slices as either CBOR nil (default) or empty container.
PR 381: Add Decoder.Skip()
to skip CBOR data item in CBOR Sequences (RFC 8742).
Most coding changes are already mentioned. Other changes include CI, comments, and docs.
Remove trailing whitespaces in .golangci.yml by @CodingVoid in https://github.com/fxamacker/cbor/pull/333
Update ci.yml by @fxamacker in https://github.com/fxamacker/cbor/pull/334
Update ci.yml by @x448 in https://github.com/fxamacker/cbor/pull/339
Update ci.yml by @x448 in https://github.com/fxamacker/cbor/pull/340
Remove default permissions from GitHub Actions workflows by @x448 in https://github.com/fxamacker/cbor/pull/341
Fix ci for stream mode branch by @fxamacker in https://github.com/fxamacker/cbor/pull/344
Create dependabot.yml by @x448 in https://github.com/fxamacker/cbor/pull/346
Bump github/codeql-action from 1 to 2 by @dependabot in https://github.com/fxamacker/cbor/pull/347
Update pull_request_template.md by @x448 in https://github.com/fxamacker/cbor/pull/350
Fix typos by @deeglaze in https://github.com/fxamacker/cbor/pull/353
Mention security assessment by NCC Group by @fxamacker in https://github.com/fxamacker/cbor/pull/358
godoc.org link in Readme should point to v2 by @jdharms in https://github.com/fxamacker/cbor/pull/361
Add Go 1.19 to ci.yml test matrix by @fxamacker in https://github.com/fxamacker/cbor/pull/363
Bump safer-golangci-lint.yml to 1.46.2 by @x448 in https://github.com/fxamacker/cbor/pull/365
Bump Go to 1.19 in ci-go-cover.yml by @x448 in https://github.com/fxamacker/cbor/pull/367
Bump safer-golangci-lint.yml to 1.49.0.1 by @x448 in https://github.com/fxamacker/cbor/pull/368
Mention 1276 repos depend on fxamacker/cbor/v2 by @x448 in https://github.com/fxamacker/cbor/pull/371
Fix EncMode example in the package comment by @creachadair in https://github.com/fxamacker/cbor/pull/375
Non-coding contibutions were made by opening notable issues that directly or indirectly improved this release.
Encode()
returning buffer to pool as default behavior.Full Changelog: https://github.com/fxamacker/cbor/compare/v2.4.0...v2.5.0-beta
This release adds two user-requested features to the decoder. It passed 3+ billion execs fuzzing before being tagged.
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.3.1...v2.4.0
IMPORTANT:
Changes to v2.3.1 include:
Full Changelog: https://github.com/fxamacker/cbor/compare/v2.3.0...v2.3.1