This is the fourth patch release for JuiceFS v1.0. It has 33 commits from 9 contributors, thanks to @zhijian-pro @SandyXSD @davies @Hexilee @tangyoupeng @molei20021 @jomenxiao @shichanglin5 @p1u3o !

Changed

• cmd/format: remove the testing directory after test (#3418)
• cmd/load: support loading metadata from an encrypted file (#3311)
• meta: recreate the client session with info if it was cleaned before (#3190, #3197)
• meta: use 'max-deletes' to control the number of background clean-up workers as well (#3227, #3401)
• meta/tikv: add background GC worker for TiKV (#3262, #3432)
• chunk: deleting an object should be idempotent (#3171)
• object: set encoding type when listing objects (#3244, #3269)
• object/s3: support using OVH region (#3133)
• object/s3: optionally turn off 'S3Disable100Continue' via url query parameter (#3228)
• deps: upgrade go version to 1.18 (#3135)
• deps: upgrade BadgerDB to v3.2103.5 (#3309)
• deps: upgrade golang.org/x/net to v0.7.0 (#3350)

Bugfix

• cmd/sync: fix the issue that password is displayed in process title (#3256, #3258)
• cmd/sync: fix the issue that sync fails when the source file is growing (#3405)
• cmd/gateway: fix the issue that a folder object is not properly detected when using s3fs on gateway (#3378)
• meta/redis: fix the issue that Redis sentinel with TLS fails hostname verification (#3194)
• meta/redis: fix the issue that client may panic on doRename when a node key is lost (#3266)
• meta/redis: fix the issue that keys may be unwatched unexpectedly in copy_file_range (#3400)
• meta/badger: fix the issue that scanKeysRange ignores begin/end (#3138)
• chunk: fix the issue that staging files may be uploaded for many times (#3157)
• vfs: fix the issue that large response may not be correct when reading from .control (#3170)
• fs: fix the issue that attribute cache is not invalidated after calling Utime (#3137)
• object/obs: fix the issue that error is not properly handled when using OBS encrypted bucket (#3199)
• object/obs&tos: fix the issue that error is not properly handled when calling RangeGet (#3270)
• object/b2: fix the issue that error is not properly handled when getting info of an empty object (#3274)
• object/sftp: fix the issue that the size of a symlinked file is not correct (#3426)
• hadoop: fix the issue that symlink is not followed as expected (091a7bff, #3165)

This is the third patch release for JuiceFS v1.0. It has 35 commits from 9 contributors, thanks to @zhijian-pro @SandyXSD @davies @tangyoupeng @Hexilee @baifachuan @neocxf @cmmp6 @zhoucheng361 !

New

• Support TOS as object storage (#2909, #2912, #2914)

Changed

• cmd: increase default value of max-deletes from 2 to 10 (#3080)
• cmd/mount: support customizing permission of cache files (#2976)
• meta: use an independent lock for session so it won't affect workloads (#3000)
• meta: limit cleanupDelayedSlices by time, which may speed up the cleanup (#3079)
• meta: force delete file data when cleaning up trash (#3119)
• meta/sql: set global lock for sqlite3 transactions to reduce conflict (#3046)
• chunk/cache: cleanup cached objects during scaning (#3113)
• object: increase buffer size from 4KB to 32KB, which can reduce CPU utilization for syscalls (#3072)
• object/encrypt: support PEM in PKCS8 (#3065)
• object/bos: support user choosing http or https (#2978)
• object/ks3: support both public and private regions (#3104)
• fuse: reduce number of max readers to save CPU utilization for hosts with many CPU cores (#3069)
• hadoop: improve the read performance when buffer size < 128 KiB (#3004)
• hadoop: add timeout for pushgateway (#3044)
• deps: upgrade tikv client-go to v2.0.2 (#2947)

Bugfix

• cmd/objbench: fix the issue that latency is wrongly calculated (#3100)
• cmd/gateway: fix the issue than HEAD a non-existent directory returns 200 (#2955)
• meta: fix the issue that client may crash when the value of slices is corrupt (#2876)
• meta: fix the issue that password may be exposed when meta engine is not available (#3003)
• meta: fix the issue that cleanup trash may run infinitely when there is a bad entry (#3032)
• meta/redis: fix the issue that slice may leak when deleting chunk (#2879)
• meta/sql: do not use the number of affected rows returned by update (#2986)
• object/sftp: fix the issue that special characters in username and password are not properly handled (#2981)
• object/s3: fix the issue that endpoint is wrongly parsed for oracle cloud (#3075)
• object/obs: fix the issue that etag is not correct when obs enables encrypted (#3098)
• hadoop: fix the issue that the running process may crash if the local juicefs-hadoop.jar is deleted (#2906)
• hadoop: fix the issue that thread may leak because emptier filesystem is not properly closed (#3090)

This is the first patch release (v1.0.1 is broken) for JuiceFS v1.0. It has 32 commits from 10 contributors, thanks to @davies @zhijian-pro @tangyoupeng @SandyXSD @timfeirg @201341 @pigletfly @dugusword @Arvintian @zwwhdls !

Changed

• cmd/sync: sync uid/gid even if the user/group does not exist (#2502)
• meta/badger: improve the performance to reset the database (#2811)
• vfs: flush data and close fd even if the close is interrupted (#2745)
• object/minio: support setting MinIO region via an environment variable (#2673)
• object/ks3: support private region for KS3 (#2812)
• gateway: support specifying umask for directories (#2445)
• hadoop: support nest juicefs-hadoop.jar in spring-boot (#2756)

Bugfix

• cmd/format: fix the issue that schema in bucket address is not properly handled (#2749)
• cmd/objbench: fix the issue that bucket is not automatically created when objbench is run with skip-functional-tests (#2773)
• cmd/sync: fix the issue that '\' is wrongly replaced for non-windows system (#2778)
• meta/pg: fix the issue that PostgreSQL does not recognize Unix domain socket properly (#2607)
• chunk/cache: fix the issue that client may crash if there are invalid files in the cache dir (#2686)
• vfs: fix the issue on Linux that entries are still visible after rmr finishes (#2776, #2838)
• vfs/backup: fix the issue that metadata backup may fail because the client cannot access /tmp (#2707)
• utils: fix the issue on Windows that logger prints color control codes (but no color) (#2567)
• compress: fix the issue that client may crash if input is empty for LZ4 decompressing (#2499)
• gateway: fix the issue that file descriptors may leak in gateway (#2661)
• hadoop: fix the issue that trash emptier may fail because of no permission (#2512)
• hadoop: fix the issue that relative paths are not properly qualified (#2696, #2709)
• hadoop: fix the issue that juicefs.umask is not properly applied when creating parent dir (#2833)
• arch: fix the issue on 32bits system that client may crash because of unaligned atomic operation (#2703)
• arch: fix the issue on 32bits system that readAheadTotal may overflow (#2726)

This is the first stable release of JuiceFS, and is an LTS version that will be maintained for 24 months.

Starting from v1.0.0-rc3 it has 73 commits from 13 contributors, thanks to @SandyXSD @zhijian-pro @xiaogaozi @zhoucheng361 @rayw000 @tangyoupeng @AIXjing @sanwan @davies @yuhr123 @timfeirg @201341 @solracsf !

New

• object/redis: support redis sentinel & cluster mode (#2368, #2383)

Changed

• cmd/gc: no warning log if no --delete specified (#2476)
• cmd/load: reset root inode to 1 if loading from a subdir (#2389)
• meta: check UUID and metadata version after setting reloaded (#2416, #2420)
• meta: reset variables at the beginning of transactions (#2402, #2409)
• meta refactor: distinguish Slice from Chunk (#2397)
• fuse: support CAP_EXPORT_SUPPORT flag (#2382)
• util: progressbar total equals current max (#2377)

Bugfix

• cmd/profile: ignore error of Scanln (#2400)
• meta: fix potential overflowed size in Fallocate (#2403)
• chunk: cleanup cache if it's added after removed (#2427)
• hadoop: do not use local uid/gid if global user/group is specified (#2433)
• hadoop: update guid and clean old ones (#2407)
• hadoop: fix make in mac m1 (#2408)

This is a test release for v1.0-GA.

JuiceFS v1.0.0-rc3 is the third release candidate for v1.0. It has 35 commits from 10 contributors, thanks to @zhijian-pro @SandyXSD @davies @tangyoupeng @sanwan @xiaogaozi @chenhaifengkeda @zhoucheng361 @201341 @Suave !

New

• Supports using unix socket for Redis (#2319)

Changed

• cmd/info: print objects for files, and add --raw option for slices (#2316)
• fuse/context: ignore interrupt within one second (#2324)

Bugfix

• cmd/info: support get trash info (#2363)
• cmd/bench: fix bench display (#2322)
• cmd/objbench: fix skip functional test (#2341)
• meta/redis: fix unlock on not-existed lock (#2325)
• object: fix skip tls verify (#2317)
• object/ks3: use virtual hosted-style for ks3 (#2349)
• object/eos: using the Path style addressing in eos (#2344)
• object/ceph: fix the arguments of register function (#2306)
• vfs&fuse: Fix stale attribute cache in kernel (#2336)
• hadoop: checksum fix for files with size close to blockSize (#2333)

JuiceFS v1.0.0-rc2 is the second release candidate for v1.0. It has 40 commits from 10 contributors, thanks to @davies @zhijian-pro @sanwan @zhoucheng361 @tangyoupeng @SandyXSD @201341 @chnliyong @solracsf @xiaogaozi !

New

• Supports using session token for object storage (#2261)

Changed

• cmd/info: make output more human friendly (#2303)
• meta: optimize output of format structure (#2250)
• meta/sql: support database without read-only transaction (#2259)
• object/webdav: replace webdav client (#2288)
• SDK: only release sdk jar (#2289)

Bugfix

• cmd: --backup-meta could be 0 to disable backup (#2275)
• cmd: fix concurrent rmr/warmup in mac (#2265)
• cmd/fsck: fix object name for Head method (#2281)
• cmd/info: Fix repeat path info when inode is 1 (#2294)
• meta: fix GetPaths for mount points with subdir specified (#2298)
• meta: don't treat trash as dangling inode (#2296)
• meta: fix index underflow in 32bit arch (#2239)
• meta/tkv: use absolute path for badgerDB as metadata engine (#2256)
• object/azure: fix azure list api and incorrect bucket URL (#2263)
• metrics: use separate metrics per volume (#2253)

JuiceFS v1.0.0-rc1 is the first release candidate for v1.0. It has 184 commits from 17 contributors, thanks to @davies @zhoucheng361 @SandyXSD @zhijian-pro @sanwan @xiaogaozi @tangyoupeng @solracsf @showjason @rayw000 @AIXjing @helix-loop @Suave @zhouaoe @chnliyong @yuhr123 @liufuyang !

Highlights

• Dumping metadata from Redis has been improved, massively reducing the memory required to below 1/20 of that before. It also relieves the memory spike of a client when doing metadata backup. Dumping metadata from SQL and TiKV within a single transaction to ensure consistency.
• Loading metadata to all engines has been improved as well. Instead of loading the whole dumped file in one step, JuiceFS will now read it in a stream, and simultaneously import metadata to the engine. This saves a lot of memory when the dumped file is huge.
• Improved stability for SQL engine under heavy workload.
• Added a new command juicefs objbench that can be used to run basic function tests and benchmarks on object storage, making sure it works as expected.

New

• Supports using SQL databases, etcd as data storage (#2003, #2009)
• Supports finding all paths of an inode with juicefs info command (#2058, #2161, #2193)
• Supports using Pyroscope to record JuiceFS profiling (#1952)
• Added progress bar for juicefs rmr and juicefs warmup commands (#2197)
• Added a new command juicefs objbench to run basic benchmarks on object storage (#2055, #2091)
• Added a new command juicefs version to print version, as an alternative to --version (#2229)

Changed

• cmd: check the range of parameters (#2195)
• cmd: eliminate panic which is triggered by missing argument (#2183)
• cmd/mount: warn about the behavior of mounting the same directory multiple times (#2141)
• cmd/warmup: support warmup from inside of a container (#2056)
• meta: add delayed slice only when chunkid > 0 (#2231)
• meta: speed up and reduce memory for loading metadata (#2142, #2148)
• meta: add a pessimistic lock to reduce conflicted transactions in the database (#2111)
• meta: limit the number of scanned files in cleanup (#2157)
• meta: limit number of files when cleanup trash (#2061)
• meta: limit the number of coroutines to delete file data (#2042)
• meta: log last error if a transaction has been ever restarted (#2172)
• meta: log session info when cleaning up a stale one (#2045)
• meta: skip updating mtime/ctime of the parent if it's updated recently (#1960)
• meta/redis: check config 'maxmemory-policy' (#2059)
• meta/redis: Speedup dump for Redis and reduce memory usage (#2156)
• meta/tkv: Speedup dump for kv storage (#2140)
• meta/tkv: dump metadata using snapshot (#1961)
• meta/tkv: use scanRange to get delayed slices (#2057)
• meta/sql: dump metadata in a single transaction (#2131)
• chunk/store: keep cache after uploading staging blocks (#2168)
• object: reload the configuration for data storage (#1995)
• object/sftp: load default private keys for sftp (#2014)
• object/oss: add user agent for oss #1992 (#1993)
• vfs: hide .control from readdir (#1998)
• gateway: clean up expired temporary files (#2082)
• SDK: package amd64 and arm64 libjfs (#2198)
• SDK: don't reuse fd in Java SDK (#2122)
• Dependency: upgrade coredns for CVE-2019-19794 (#2190)
• Dependency: upgrade azblob sdk (#1962)
• meta: keep valid utf8 in dumped JSON (#1973)
• SDK: mvn shade some dependency to avoid class conflict (#2035)
• meta: truncate trash entry name if it's too long (#2049)
• meta/sql: use repeatable-read for transaction (#2128)

Bugfix

• cmd: fix not showing arguments for commands without META-URL (#2158)
• cmd/sync: fix sync lost file (#2106)
• cmd/warmup: fix warmup on read-only mount point (#2108)
• meta: skip updating sliceRef if id is 0 (#2096)
• meta: fix update xattr with the same value (#2078)
• meta/redis: handle lua result from Redis v7.0+ (#2221)
• meta/sql: fix dump with more than 10000 files (#2134)
• meta/sql: one transaction in SQLite at a time (#2126)
• meta/sql: fix rename with Postgres with repeatable read (#2109)
• meta/sql: fix deadlock in PG (#2104)
• meta/sql: ignore error about duplicated index (#2087)
• meta/sql: read database inside transaction (#2073, #2086)
• meta/sql: retry transaction on duplicated entry and concurrent update (#2077)
• meta/sql: fix the deadlock in rename (#2067)
• meta/sql: retry transaction for duplicate key in PG (#2071)
• meta/sql: fix update query in SQL transaction (#2024)
• meta/tkv: fix value of delSliceKey (#2054)
• meta/tkv: upgrade TiKV client to 2.0.1 to fix nil connection (#2050)
• chunk/store: fix stats of cached space in writeback mode (#2227)
• object: delete should be idempotent (#2034)
• object/file: Head of file should return File (#2133)
• object/s3: check prefix and marker with returned keys from S3 (#2040)
• object/prefix: fix with prefix returning nil error for unsupported ops (#2021)
• object/sftp: fix auth of sftp with multiple keys (#2186)
• object/sftp: fix prefix of sftp, support ssh-agent (#1954)
• vfs/backup: skip cleanup if list failed (#2044)
• SDK: handle atomic rename exception (#2192)

JuiceFS v1.0.0-beta3 is the third beta release for v1.0. It has 247 commits from 22 contributors, thanks to @SandyXSD @zhoucheng361 @davies @zhijian-pro @yuhr123 @sanwan @AIXjing @rayw000 @xiaogaozi @Suave @showjason @tangyoupeng @201341 @solracsf @guo-sj @chnliyong @DeanThompson @zwwhdls @wph95 @lidaohang @sjp00556 @DEvmIb !

Highlights

• Supports etcd as a new metadata engine. It can be a handy choice when you only need a small volume but cares more about the data availability and persistence.
• Supports Redis Cluster and other compatible services (Amazon MemoryDB for Redis) as metadata engines.
• When using SQL metadata engines, file names not encoded by UTF-8 can now be properly handled after manual modification to the table schema, see details.
• A new session management format is introduced. Old clients are unable to detect sessions with version 1.0.0-beta3 or higher via juicefs status or juicefs destroy command, see details.
• If trash is enabled, compacted slices are kept as well in case they are needed to recover file data. These slices will be cleaned up automatically after trash-days, and can be deleted manually via juicefs gc command.
• A lot of improvements have been made to juicefs sync command.
• A lot of protection checks against unintentional misuse have been added.

New

• Supports etcd as metadata engine (#1638)
• Supports Redis in cluster mode while using one slot for each file system (#1696)
• Supports handling file names not encoded by UTF-8 for SQL metadata engines (#1762)
• Supports TLS when using TiKV as metadata engine or object storage (#1653, #1778)
• Supports Oracle Object Storage as data storage (#1516)
• Supports setting umask for S3 Gateway (#1537)
• Java SDK now supports pushing metrics to Graphite (#1586)
• Added a new option --heartbeat for the mount command to adjust heartbeat interval (#1591, #1865)
• Added many improvements for sync command to make it more handy (#1554, #1619, #1651, #1836, #1897, #1901)
• Added a new option --hash-prefix for the format command to add a hashed prefix for objects (#1657)
• Added a new client option --storage to allow customized storage type (#1912)

Changed

• compacted slices will be kept for trash-days if trash is enabled (#1790)
• cmd: support using integer for duration flags (#1796)
• cmd: use homedir as default working directory for non-root users (#1869)
• cmd/format: create a uuid object in the target bucket (#1548)
• cmd/dump&load: dump load behavior supports non-ascii characters (#1691)
• cmd/dump: omit empty value in dumped JSON (#1676)
• cmd/dump: remove secret key (#1569)
• meta: encrypt the secret-key and encrypt-key in setting (#1562)
• meta: create subdir automatically (#1712)
• meta: specify the format field preventing update (#1776)
• meta: escape meta password from env (#1879)
• meta/redis: check redis version (#1584)
• meta/redis: use smaller retry backoff in sentinel mode (#1620)
• meta/redis: retry transaction for connection error or EXECABORT (#1637)
• meta/sql: retry transaction after too many connections (#1876)
• meta/sql: add primary key for all tables (#1913, #1919)
• meta&chunk: Set max retries of meta & chunk according to the config io-retries (#1713, #1800)
• chunk: limit number of upload goroutines (#1625)
• chunk/store: limit max retry for async upload as well (#1673)
• object/obs: Verify Etag from OBS (#1715)
• object/redis: implement listAll api for redis (#1777)
• fuse: automatically add ro option if mount with --read-only (#1661)
• vfs/backup: reduce the limit for skipping backup (#1659)
• sync: reduce memory allocation when write into files (#1644)
• SDK: use uint32 for uid,gid (#1648)
• SDK: handle removeXAttr return code (#1775)
• Dependency: switch to Go 1.17 (#1594)
• Dependency: fix jwt replace (#1534)
• Dependency: upgrade golang-cross version to v1.17.8 (#1539)
• Dependency: upgrade tikv to v2.0.0 (#1643)
• Dependency: reduce dep from minio (#1645)
• Dependency: upgrade gjson to 1.9.3 (#1647)
• Dependency: upgrade sdk for object storage (#1665)
• Dependency: upgrade qiniu sdk (#1697)

Bugfix

• cmd/format: fix setting quota (#1658)
• cmd/mount: fix parsing of cache dir (#1758)
• cmd/warmup: fix handling of relative paths (#1735)
• cmd/sync: fix sync command not working when destination is webdav (#1636)
• cmd/gateway: fix s3 gateway DeleteObjects panic (#1527)
• meta: forbid empty name for dentry (#1687)
• meta: lock counters when loading entries (#1703)
• meta: fix snap not released if error occurs when dumping meta (#1669)
• meta: don't update parent attribute if it's a trash directory (#1580)
• meta/redis: fix loading large directory into Redis (#1858)
• meta/redis: update used space/inodes in memory whenever needed (#1573)
• meta/sql: use upsert to update xattr for PG (#1825)
• meta/sql: split insert batch (#1831)
• meta/sql: fix wrong result from scanning SQL database (#1854)
• chunk/cache: Fix read disk_cache not existed remove cache key (#1677)
• object: fallback to List only if ListAll is not supported (#1623)
• object/b2: check returned bucket from B2 (#1745)
• object/encrypt: fix parse rsa from pem (#1724)
• object/encrypt: Add JFS_RSA_PASSPHRASE environment variable prompt information (#1706)
• object/sharding: fix ListAll returning invalid objects (#1616)
• object/ceph: fix listAll hangs if there are many objects (#1891)
• vfs: write control file asynchronously (#1747)
• vfs: fix getlk in access log (#1788)
• sync: Fix copied and copiedBytes (#1801)
• utils: fix the problem that the progress bar loses the log (#1756)
• SDK: rename libjfs atomic (#1939)

JuiceFS v1.0.0-beta2 is the second beta release for v1.0. It has 150+ commits from 16 contributors, thanks to @SandyXSD @zhijian-pro @yuhr123 @xiaogaozi @davies @sanwan @AIXjing @Suave @tangyoupeng @zwwhdls @201341 @zhexuany @chnliyong @liufuyang @rayw000 @fredchen2022 !

New

• Supports BadgerDB (an embeddable key-value database) as metadata engine (#1340)
• Supports WebDAV protocol to access JuiceFS files (#1444)
• Supports read-only clients connecting to a Redis Sentinel-controlled replica (#1433)
• Added version control of metadata and clients (#1469)
• Added categories and long descriptions for all commands (#1488, #1493)
• Added a new option no-bgjob for service commands to disable background jobs like clean-up, backup, etc. (#1472)
• Added metrics for number of rawstaging blocks (#1341)
• Added cross-platform compile script (#1374)

Changed

• cmd: help command is removed; use --help/-h flag to get help information (#1488)
• cmd: print usage if not enough args (#1491)
• cmd/format: only try to create bucket when it really doesn't exist (#1289)
• cmd/format: prevend reusing object storage when formating a volume (#1420, #1449)
• cmd/destroy: show information of active sessions (#1377)
• cmd/config: add sanity check of new values (#1479)
• cmd/umount: ignore error from umount (#1425)
• meta: add progress bar for CompactAll (#1317)
• meta: hide password in meta-url (#1333, #1361)
• meta: use META_PASSWORD env and omit unnecessary characters (#1388)
• meta: limit the number when scanning sessions/files (#1397)
• meta: limit the number of clients running background cleanup jobs (#1393)
• meta: continue dump when encountering non-fatal errors (#1462)
• meta/sql: increase max idle connections to CPU*2 for SQL engine (#1443)
• object: set ContentType when putting object (#1468)
• object: support skip https certificate verify (#1453)
• object/s3: format support pvc link (#1382)
• object/qingstor: support private cloud and replace the sdk repository (#1303)
• vfs: don't do backup for read-only clients (#1435)
• vfs: add BackupMeta in config (#1460)
• utils: log now contains the exact line of caller, and is colorized in terminal (#1404, #1318, #1312)
• utils: simplify the usage of progress bar (#1325)
• utils: add SleepWithJitter to reduce collision of background jobs (#1412)
• SDK: upgrade Hadoop common to 3.1.4 (#1411)
• SDK: java sdk push gateway support multiple volumes (#1492)
• Other: update chart move it to a standalone repository (#1281, #1336, #1348)
• Improves documentation and coverage of tests

Bugfix

• cmd: fix buffer-size in gc and fsck (#1316)
• cmd/bench: convert PATH to absolute path (#1305)
• meta: return EROFS as soon as possible (#1477)
• meta/redis: fix leaked inodes in Redis (#1353)
• meta/tkv: fix divide by zero error when dumping meta (#1369)
• meta/tikv: fix scan of tikv, limiting the upperbound (#1455)
• meta/memkv: fix scanKeys, returning a sorted list (#1381)
• meta/sql: delete warning message for empty directory (#1442)
• meta/sql: fix return value of mustInsert (#1429)
• vfs: fixed deadlock when truncate a released file handle. (#1383)
• vfs/trash: fix access to trash dir (#1356)
• vfs/backup: skip dir objects when scanning meta backups (#1370)
• vfs/backup: fix incorrect inode number when using subdir (#1385)
• utils: fix the contention between progress bar and logger (#1436)
• Windows: fix rename fails because the chunk file is still open (#1315)
• Windows: fix mkdir on windows platform (#1327)
• SDK: hadoop: fix umask apply (#1338, #1394)
• SDK: hadoop: fix libjfs.so load bug (#1458)
• other: fix legend of "Operations" panel in Grafana template (#1321)