Dvid Versions Save

A backward-compatible release that can be used for previously created databases. A future 2.0 version DVID will have a metadata overhaul to remove unused legacy datatypes and simplify management.

Changes include:

• Many neuronjson datatype improvements.
  • Allows multiple in-memory databases for both committed and branch HEAD versions of neuronjson datatype.
  • Automatic _user and _time fields that won't automatically change if field values aren't changed even if replacement requested.
• Improvements in handling of proxied uint8blk grayscale image volumes from neuroglancer precomputed data via gridstore interface.
• Allow simpler tag-style identifiers as assigned UUID (e.g., v1.0)
• Allow disabling /split operations via configuration
• Much faster /sparsevol operations if requesting supervoxels.
• Faster range query scans in /blocks for annotation datatype

Significant changes:

• New neuronjson datatype that is tailored for a small JSON doc (value) for each neuron ID (key). It is a replacement for neuron annotations that had previously been held within keyvalue instances and uses in-memory store for the HEAD version and persists data to chosen backend.
• Upgrade from Badger v2 to v3. This will require you to upgrade your Badger database directories if you have an older version.
• Switch to Go modules from older "go get" install scripts.
• Simplified conda builds and release due to switch to Go modules. (@stuarteberg)
• Support for AWS S3 in the ngprecomputed backend. (Contributed by @pgunn, Flatiron Institute)
• Expanded number of versions for in-memory mapping in labelmap datatype. Limit before was 256 versions, now expanded to uint32.
• Ability to renumber labels in labelmap datatype, as well as keeping track of split or renumbered labels by inserting 0 into in-memory mapping.
• Can now specify Xth parent of a branch leaf using uuid:branch~X for UUID.
• Lots of fixes.

• fix labelmap /history and improve documentation for it
• add neuronjson datatype with Google Firestore persistence
• add optimized ingest supervoxel blocks
• fix bug in install-developer-dependencies
• Kafka-related changes:
  • switch to GoCloud SDK pubsub for Kafka support
  • convert to sarama kafka driver; send Key for compacted log issue?
  • use async producer for kafka
  • remove deprecated kafka properties from testing
• Optimizations for concurrent interactions with backend for labelmap blocks (leveldb)
  • remove Cgo tracking from read hotpaths; not necessary
  • simplify active CGo write tracking by removing channel in favor of atomic write
  • restrict concurrency in labelmap /specificblocks to transcoding

• Remove unused, old storage drivers
• Various build script changes
• Now compiled with go-1.16

Improved tooling for repository handling and fixes.

Low-memory mode disabled

Previous binary release tarballs (from v0.9.3 through v0.9.10) were built with DVID_LOW_MEMORY=1, which forced badger databases to operate in a special low-memory mode. This is not desirable for production uses. Release builds published here on our github releases page had sub-optimal performance. Our build procedure has changed to avoid using DVID_LOW_MEMORY in release builds.

Other changes

• add update of max label on label index ingestion; fixes #352
• return 404 instead of 204 for non-existant key in keyvalue instance; fixes #353
• allow changing branch names as part of flatten-metadata rpc command
• fix possible race condition on labelsz test, which might lead to test fails on resource-limited build machines

• add /listlabels to labelmap
• do fsync on logs to guarantee writes in exchange of throughput
• add public versions and Google id auth
• add ability to set next labels in labelmap
• add more efficient /indices-compressed to labelmap
• force directory creation for json mutation log
• don't let server start if mutations jsonstore can't be created when specified
• By default, release distribution tarball uses dvid-web-console 2.1.7
• allow repo-level rename/new instance rpc on locked node

Note: No Mac tarball yet.

• allow setting readonly and fullwrite modes in TOML config
• add error check for analyze-block command
• add scale option for analyze-block cmd
• add json mutation logging for other labelmap ops
• fix jpeg uncompression on ngprecomputed
• add unsharded ng precomputed support; allow auto instances for ngprecomputed
• fix rounding error on ng precomputed indexing
• add POST /api/repo/{uuid}/info
• further error checking on ngprecomputed block encodes
• make ngprecomputed more resilient to various JPEG encoding styles
• revert imageblk GetVoxels before gridstore support
• fixes number of issues for ngprecomputed driver including clipped chunks
• try another approach to wildcard origin handler
• fix bug introduced patching issue #347
• fix initialization of ngprecomputed extents
• handle metadata and improve doc for ngprecomputed uint32blk
• fix inconsistent CORS handling based on config

• JSON marshaling of data instances' versioned props in repos/info assumes leaf of master branch
• fix buggy extents update; fixes #301
• add POST /extents to labelmap; fixes #344
• add documentation for POST /extents to labelmap
• add uncompressed data return in subvolblocks endpoint; fixes #347
• minor init logging formatting change
• allow admin-enabled mutations to bypass read-only mode
• upgrade to lz4 1.9.2

• fix annotation block size at 64. Note this is dangerous if you have legacy annotation at 32
• add POST /blocks to annotation
• fixes to flatten-metadata
• add admin token querystring to allow selective full write
• add version-specific migration and full test suite
• allow version range for mutation log flattening
• add migrate-batch command
• enhance flatten metadata command to multi-version use
• fix multiversion flattening
• remove debug that can fill logs for large queries
• decouple CORS from Authorization
• fix dag duplication on version subset; add test
• remove transactionDB/gbucket mutex and patch
• prevent any metadata writes in read-only mode
• fix copy kv pairs to also transmit version 0 metadata per data instance
• prevent package doc in generated protobuf
• allow wildcard auth and block based on HTTP Method
• add annotation /all-elements request
• add auth file reload without reboot
• fix branch ancestry calc, initial mapping load
• add labelmap mutation JSON log
• prevent absolute path conversion from creating config entries
• add ngprecomputed backend for static Neuroglancer Precomputed GCS store
• add /keyrangevalues endpoint for keyvalue data type
• add support for confluent cloud kafka auth
• add kakfa topic creation
• update confluent kafka libs
• add auth option for just token verification
• update brainmaps API for googlevoxels
• allow CORS to be shutoff in case of nginx fronting
• return better error responses for /api/server/token
• allow wildcard access for CORS issues if auth turned off
• Ship both versions of the dvid console, leave the old version active by default