Cbor Versions Save

CBOR codec (RFC 8949) with CBOR tags, Go struct tags (toarray, keyasint, omitempty), float64/32/16, big.Int, and fuzz tested billions of execs.

v2.6.0

3 months ago

This release adds important new features, optimizations, bug fixes, and 3 new contributors. Fuzz tests passed 5+ billion execs for CBOR encoding and decoding functions. Minimum officially supported Go is bumped to Go 1.17.

⭐ Features and Optimizations

🚀 Notable Optimizations

  • Improve memory allocs 🗜️ and speed 🚀 of encoding maps by using Go 1.18-1.20 features by @dinhxuanvu in https://github.com/fxamacker/cbor/pull/468

    📊 Benchmarks

    benchstat results provided by @dinhxuanvu

    Screenshot of CBOR benchmarks

    NOTE: Go 1.17 is the minimum version supported by v2.6.0. Go 1.20+ will use optimizations introduced by PR 468.

🛠 Improvements

🐞 Bug Fixes

📖 Docs and CI

New Contributors

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0...v2.6.0

v2.6.0-rc.0

3 months ago

⭐ Features and Optimizations

🚀 Notable Optimizations

  • Improve memory allocs 🗜️ and speed 🚀 of encoding maps by using Go 1.18-1.20 features by @dinhxuanvu in https://github.com/fxamacker/cbor/pull/468

    📊 Benchmarks

    benchstat results provided by @dinhxuanvu

    Screenshot of CBOR benchmarks

    NOTE: Go 1.17 is the minimum version supported by v2.6.0. Go 1.20+ will use optimizations introduced by PR 468.

🛠 Improvements

🐞 Bug Fixes

📖 Docs and CI

New Contributors

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0...v2.6.0-rc.0

v2.5.0

9 months ago

This release adds important new features, optimizations, bug fixes, and 8 new contributors. Fuzz tests passed 5+ billion execs for CBOR encoding and decoding functions.

Notable changes are split into 3 categories.

⭐ Notable Changes to Review Before Upgrading

These new features and bug fixes were cherry-picked to highlight for review (for projects using older version).

  • PR 370: Add SimpleValue type to more fully support CBOR Simple Values, including values not assigned by IANA and...
  • PR 376: Add ByteString type to support CBOR maps with byte string keys because Go doesn't allow []byte as map keys and...
  • PR 379: Make Decoder.Decode() return io.ErrUnexpectedEOF instead of io.EOF on EOF if current CBOR data item is incomplete.
  • PR 380: Make Unmarshal() and Valid() return cbor.ExtraneousDataError (instead of ignoring extraneous data if any remain).
  • PR 387: Retry in Decoder if io.Reader's Read() returns 0 bytes read with nil error. Add tests to get 100% code coverage for stream.go.

⭐ More Notable Changes

  • PR 342: Add DecOptions.UTF8 to decode invalid UTF-8. Default is unchanged (reject invalid UTF-8 and return error).
  • PR 355 Allow MaxNestedLevels setting to be configured up to 65535.
  • PR 352, 377: Add EncOptions.NilContainersMode to encode nil Go maps and slices as either CBOR nil (default) or empty container.
  • PR 381: Add Decoder.Skip() to skip CBOR data item in CBOR Sequences (RFC 8742).
  • PR 386 Add functions for Extended Diagnostic Notation (RFC 8610 Appendix G).
  • PR 398 Add UnmarshalFirst() to decode CBOR data item and also return remaining bytes.
  • PR 400 Deprecate Valid and add Wellformed to replace it.
  • PR 402 Add UnmarshalFirst() to DecMode interface.
  • PR 412 Add Decoder.Buffered to return remaining data in buffer.

🚀 Notable Changes Affecting Speed or Memory Use

  • PR 335: Reuse underlying array if RawMessage has sufficient capacity.
  • PR 382: Return buffer to pool in Encode(). It adds a bit of overhead to Encode() but NewEncoder().Encode() is faster and uses less memory.

Benchmark comparison for PR 382 shows significant improvement in speed and memory use.

:rocket: (click to expand) :rocket:

Benchmarks provided by @x448 for NewEncoder().Encode() optimized in PR 382.

$ benchstat bench-v2.4.0.log bench-f9e6291.log 
goos: linux
goarch: amd64
pkg: github.com/fxamacker/cbor/v2
cpu: 12th Gen Intel(R) Core(TM) i7-12700H
                                                     │ bench-v2.4.0.log │  bench-f9e6291.log                  │
                                                     │      sec/op      │   sec/op     vs base                │
NewEncoderEncode/Go_bool_to_CBOR_bool-20                   236.70n ± 2%   58.04n ± 1%  -75.48% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20         238.00n ± 2%   63.93n ± 1%  -73.14% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20          238.65n ± 2%   64.88n ± 1%  -72.81% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20               242.00n ± 2%   63.00n ± 1%  -73.97% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20               245.60n ± 1%   68.55n ± 1%  -72.09% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20                 243.20n ± 3%   68.39n ± 1%  -71.88% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20                 563.0n ± 2%    378.3n ± 0%  -32.81% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20       2.043µ ± 2%    1.906µ ± 2%   -6.75% (p=0.000 n=10)
geomean                                                    349.7n         122.7n       -64.92%

                                                     │ bench-v2.4.0.log │    bench-f9e6291.log                │
                                                     │       B/op       │    B/op     vs base                 │
NewEncoderEncode/Go_bool_to_CBOR_bool-20                     128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20           128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20            128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20                 128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20                 128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20                   128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20                   128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20         544.0 ± 0%   416.0 ± 0%   -23.53% (p=0.000 n=10)
geomean                                                      153.4                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

                                                     │ bench-v2.4.0.log │    bench-f9e6291.log                │
                                                     │    allocs/op     │ allocs/op   vs base                 │
NewEncoderEncode/Go_bool_to_CBOR_bool-20                     2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20           2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20            2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20                 2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20                 2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20                   2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20                   2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20         28.00 ± 0%   26.00 ± 0%    -7.14% (p=0.000 n=10)
geomean                                                      2.782                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

What's Changed

All merged pull requests are listed here (including the already highlighted ones).

(click to expand)

Changes to Code

Most coding changes here were already mentioned.

Changes to CI, Comments, and Docs

New Contributors (alphabetically)

Non-coding contibutions were made by opening notable issues that directly or indirectly improved this release.

  • @burdiyan opened issue and followups that led to Encode() returning buffer to pool as default behavior.
  • @espoal opened issue and followups that led to adding Skip() feature for CBOR Sequences (RFC 8742).
  • @immesys opened multiple issues and provided helpful feedback in addition to merged PRs.
  • @qmuntal opened issue and @x448 added feedback that led to improved support for CBOR Simple Values.
  • @x448 opened multiple issues and provided helpful feedback in addition to merged PRs.
  • @zensh opened multiple issues and provided helpful feedback in addition to merged PRs.

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.4.0...v2.5.0

v2.5.0-beta5

10 months ago

v2.5.0-beta5 is fuzz tested and production quality, but docs need to be updated before releasing v2.5.0. IMPORTANT: Please view these release notes (esp. v2.5.0-beta) before upgrading from v2.4 or earlier:

  • v2.5.0-beta4 - Bugfix for Diagnose to return io.EOF on empty data.
  • v2.5.0-beta3 - Add functions: Diagnose, DiagnoseFirst, UnmarshalFirst, Wellformed.
  • v2.5.0-beta2 - Bugfix to retry in Decoder if io.Reader's Read() returns 0 bytes read with nil error.
  • v2.5.0-beta - Notable improvements, optimizations, bugfixes, and 8 new contributors!

What's Changed since v2.5.0-beta4

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta4...v2.5.0-beta5

v2.5.0-beta4

11 months ago

Features already present in v2.4 are release quality. Newly added functions require more fuzzing and docs.
IMPORTANT: Please view these release notes (esp. v2.5.0-beta) before upgrading from v2.4:

  • v2.5.0-beta3 - Add Diagnose(), DiagnoseFirst(), UnmarshalFirst(), Wellformed().
  • v2.5.0-beta2 - Bugfix and release notes include benchmark comparison of v2.5.0-beta2 vs v2.4.0.
  • v2.5.0-beta - Notable improvements, optimizations, bugfixes, and 8 new contributors!

What's Changed Since v2.5.0-beta3

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta3...v2.5.0-beta4

v2.5.0-beta3

1 year ago

This release adds 4 new functions: UnmarshalFirst, Diagnose, DiagnoseFirst, and Wellformed. In addition Valid function is deprecated (use Wellformed instead).

  • UnmarshalFirst decodes first CBOR data item and returns remaining bytes. This is useful for CBOR Sequences (RFC 8742) and also because Unmarshal was fixed in v2.5.0-beta to return ExtraneousDataError if there are remaining bytes.
  • Diagnose and DiagnoseFirst returns human-readable Extended Diagnostic Notation (RFC 8610 Appendix G) for the given CBOR encoding. These are useful for debugging and allows more user-friendly error logging.

What's Changed (since v2.5.0-beta2)

Changes to CI and Documentation

Special Thanks

Thank you @immesys, @zensh, and @x448 for your contributions!

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta2...v2.5.0-beta3

v2.5.0-beta2

1 year ago

There are fewer changes in v2.5.0-beta2 compared to v2.5.0-beta. Speed and memory improvements since v2.4.0 are from v2.5.0-beta. Tests cover 98.5% and reached 100% coverage for stream.go (e.g. Decoder using the provided io.Reader).

What's Changed Since v2.5.0-beta

Go language recommends that Read implementations for io.Reader avoid returning 0 with nil error. It also recommends that callers receiving 0 and nil from Read should treat it as if nothing happened. This edge case is handled properly now.

@x448 shared benchmarks comparison of v2.4.0 vs v2.5.0-beta2. Performance improvements are from v2.5.0-beta.

Comparison of v2.5.0-beta2 vs v2.4.0

$ benchstat benchNewEncoderEncode-v2.4.0.log benchNewEncoderEncode-f9e6291.log 
goos: linux
goarch: amd64
pkg: github.com/fxamacker/cbor/v2
cpu: 12th Gen Intel(R) Core(TM) i7-12700H
                                                     │ benchNewEncoderEncode-v2.4.0.log │  benchNewEncoderEncode-f9e6291.log  │
                                                     │              sec/op              │   sec/op     vs base                │
NewEncoderEncode/Go_bool_to_CBOR_bool-20                                   236.70n ± 2%   58.04n ± 1%  -75.48% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20                         238.00n ± 2%   63.93n ± 1%  -73.14% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20                          238.65n ± 2%   64.88n ± 1%  -72.81% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20                               242.00n ± 2%   63.00n ± 1%  -73.97% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20                               245.60n ± 1%   68.55n ± 1%  -72.09% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20                                 243.20n ± 3%   68.39n ± 1%  -71.88% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20                                  563.0n ± 2%   378.3n ± 0%  -32.81% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20                        2.043µ ± 2%   1.906µ ± 2%   -6.75% (p=0.000 n=10)
geomean                                                                     349.7n        122.7n       -64.92%

                                                     │ benchNewEncoderEncode-v2.4.0.log │    benchNewEncoderEncode-f9e6291.log    │
                                                     │               B/op               │    B/op     vs base                     │
NewEncoderEncode/Go_bool_to_CBOR_bool-20                                     128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20                           128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20                            128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20                                 128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20                                 128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20                                   128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20                                   128.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20                         544.0 ± 0%   416.0 ± 0%   -23.53% (p=0.000 n=10)
geomean                                                                      153.4                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

                                                     │ benchNewEncoderEncode-v2.4.0.log │    benchNewEncoderEncode-f9e6291.log    │
                                                     │            allocs/op             │ allocs/op   vs base                     │
NewEncoderEncode/Go_bool_to_CBOR_bool-20                                     2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_uint64_to_CBOR_positive_int-20                           2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_int64_to_CBOR_negative_int-20                            2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_float64_to_CBOR_float-20                                 2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]uint8_to_CBOR_bytes-20                                 2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_string_to_CBOR_text-20                                   2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_[]int_to_CBOR_array-20                                   2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
NewEncoderEncode/Go_map[string]string_to_CBOR_map-20                         28.00 ± 0%   26.00 ± 0%    -7.14% (p=0.000 n=10)
geomean                                                                      2.782                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.5.0-beta...v2.5.0-beta2

v2.5.0-beta

1 year ago

This release is larger than usual because it has been a year since last release. It adds new features, bug fixes, and 8 new contributors.

All unit tests pass (98.4% coverage). Fuzz tests passed hundreds of millions of execs. Fuzzing will continue until v2.5.0 is tagged.

Notable Changes to Review Before Upgrading

These new features and bug fixes were cherry-picked to highlight for review (for projects using older version in production).

  • PR 370: Add SimpleValue type to more fully support CBOR Simple Values, including values not assigned by IANA and...

  • PR 376: Add ByteString type to support CBOR maps with byte string keys because Go doesn't allow []byte as map keys and...

  • PR 379: Make Decoder.Decode() return io.ErrUnexpectedEOF instead of io.EOF on EOF if current CBOR data item is incomplete.

  • PR 380: Make Unmarshal() and Valid() return cbor.ExtraneousDataError (instead of ignoring extraneous data if any remain).

Notable Changes Affecting Speed or Memory Use

  • PR 335: Reuse underlying array if RawMessage has sufficient capacity.

  • PR 382: Return buffer to pool in Encode(). It adds a bit of overhead to Encode() but NewEncoder().Encode() is a lot faster and uses less memory.

Other Notable Changes:

  • PR 342: Add DecOptions.UTF8 to decode invalid UTF-8. Default is unchanged (reject invalid UTF-8 and return error).

  • PR 355 Allow MaxNestedLevels setting to be configured up to 65535.

  • PR 352, 377: Add EncOptions.NilContainersMode to encode nil Go maps and slices as either CBOR nil (default) or empty container.

  • PR 381: Add Decoder.Skip() to skip CBOR data item in CBOR Sequences (RFC 8742).

What's Changed

Most coding changes are already mentioned. Other changes include CI, comments, and docs.

(click to expand)

All Changes to Code

Changes to CI, Comments, and Docs

New Contributors (alphabetically)

Non-coding contibutions were made by opening notable issues that directly or indirectly improved this release.

  • @burdiyan opened issue and followups that led to Encode() returning buffer to pool as default behavior.
  • @espoal opened issue and followups that led to adding Skip() feature for CBOR Sequences (RFC 8742).
  • @qmuntal opened issue and @x448 added feedback that led to improved support for CBOR Simple Values.
  • @x448 opened multiple issues and provided helpful feedback in addition to his merged PR.
  • @zensh opened multiple issues and provided helpful feedback in addition to his merged PR.

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.4.0...v2.5.0-beta

v2.4.0

2 years ago

This release adds two user-requested features to the decoder. It passed 3+ billion execs fuzzing before being tagged.

What's Changed

Special Thanks

  • @lukseven for opening issue #303 and proposing a solution
  • @Gui-Yom for issue #301

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.3.1...v2.4.0

v2.3.1

2 years ago

IMPORTANT:

  • This release fixes an important typo in README and omission in CONTRIBUTING.
  • No changes to code outside _test.go files.
  • Changes to non-test files are limited to comments.
  • Next release (v2.4.0) started fuzz testing and is expected to be tagged within 1-2 weeks.

Changes to v2.3.1 include:

  • Fix typo in docs (example code snippet) that can cause bugs. Thanks @herrjemand!
  • Update CONTRIBUTING to mention signing requirements. Thanks @lukseven and @x448!
  • Update README. Thanks @x448 and @rumpelsepp!
  • Update ci.yml to use Go 1.17.x. Thanks @x448!
  • Add Revive as a lint checker.
  • Cleanup lint messages in _test.go files
  • Cleanup lint messages in non-test files if the changes are limited to comments (no actual coding changes).

Full Changelog: https://github.com/fxamacker/cbor/compare/v2.3.0...v2.3.1