Littlefs Versions Save

1bc1493 Tweaked on-disk config comments for consistency 01b6a47 Extended test_alloc to test inferred block_count 749a456 Fix DivideByZero exception when filesystem is completely full.

7873d81 Fixed memory leak in emubd's out-of-order write emulation fc2aa33 Fixed issue with exhaustive + out-of-order powerloss testing 6352185 Fixed sync issue where data writes could appear before metadata writes f2a6f45 Added out-of-order write testing to emubd 2752d8c add nim-littlefs to readme ddbfcaa Fixed synthetic move underflows in lfs_dir_get

Itsy-bitsy breaking changes:

• lfs_rename now returns LFS_ERR_NOTDIR if the source is a regular file and the destination is a directory. This better aligns with POSIX.

• Defining LFS_FILE_MAX > 2147483647 is no longer supported. It's unclear if this ever worked, and at least half the file API was unusable in this state.

What's new?

• lfs_fs_gc can now compact metadata (#913)

  This is controlled by the new optional compact_thresh configuration option. mdirs above compact_thresh will be eagerly compacted during lfs_fs_gc, making it less likely (but still possible) for compactions to occur during file writes.

  By default, compact_thresh=0, littlefs will compact mdirs > ~88% block_size. This value may change in the future.

  compact_thresh=-1 disables mdir compaction during lfs_fs_gc.

  Note that compact_thresh has no effect on mdir compaction outside of lfs_fs_gc. mdirs must compact when full.

• Added inline_max, which allows explicit user control over the size of inlined files (#914)

  Decreasing inline_max may improve some metadata-related performance around mdir traversal. But decreasing inline_max may also hurt metadata-related performance around mdir compaction and increase storage consumption. It is up to users to determine the best value per-application.

  By default, inline_max=0, uses the largest possible inline_max. This matches the previous behavior and is probably a good default for most use cases.

  inline_max=-1 disables inlined files.

• Added easier util overrides for common system functions: LFS_MALLOC, LFS_FREE, LFS_CRC (#909)

  These can now be overridden with simple defines:

  -DLFS_MALLOC=my_malloc
  -DLFS_FREE=my_free
  -DLFS_CRC=my_crc
  

  Note: Overriding LFS_CRC with a non-CRC32 checksum is discouraged. This is only intended for hardware acceleration, etc.

• Thanks to @tomscii, lfs_rename now returns LFS_ERR_NOTDIR if the source is a regular file and the destination is a directory, which better matches POSIX (#917)

• Thanks to @BrianPugh, defining overridable limits (LFS_FILE_MAX, LFS_NAME_MAX, LFS_ATTR_MAX) to incompatible values will now result in a compile error (#886)

• Relaxed lookahead buffer alignment to only be byte-aligned (#912)

• Relaxed lfs_malloc alignment requirements to only be byte-aligned (#912)

Changes

42910bc Bumped minor version to v2.9 a70870c Renamed internal functions _raw* -> * 8b8fd14 Added inline_max, to optionally limit the size of inlined files b5cd957 Extended lfs_fs_gc to compact metadata, compact_thresh f522ed9 Added tests over rename type errors 4f32738 Fix return value of lfs_rename() 6691718 Restricted LFS_FILE_MAX to signed 32-bits, <2^31, <=2147483647 1fefcbb Rearranged compile-time constant checks to live near lfs_init 6056767 Relaxed alignment requirements for lfs_malloc 897b571 Changed CI to just run on ubuntu-latest b1b10c0 Relaxed lookahead buffer alignment 1f9c3c0 Reworked the block allocator so the logic is hopefully simpler 7b68441 Renamed a number of internal block-allocator fields 9a620c7 Added LFS_CRC, easier override for lfs_crc a0c6c54 Added LFS_MALLOC/FREE, easier overrides for lfs_malloc/free c531a5e Replace erroneous LFS_FILE_MAX upper bound 4294967296 to 4294967295 8f9427d Add value-range checks for user-definable macros

Sponsors

A special thanks to littlefs's sponsors: @nedap, @kmetabg, @fusedFET

Changes

99b84ee Update DESIGN.md, fix minor typo e91a29d Fixed struct sizes missing from generated release notes b9b95ab Increase threshold for superblock expansion from ~50% -> ~88% full 10bcff1 Update DESIGN.md minor typo

Contributors

A special thanks to all who proposed PRs: @zchen24

Sponsors

And a special thanks to littlefs's sponsors: @nedap, @kmetabg, @fusedFET

8f3f32d Added -Wmissing-prototypes 92fc780 lfs_fs_raw* functions should be static

This release brings in some useful community-driven features.

What's new?

• Thanks to @BrianPugh, littlefs no longer needs the block_count when mounting an existing filesystem (#866)

  Simply set block_count=0 in your configuration, and littlefs will automatically determine the block_count based on what's on disk. This allows you to mount a filesystem of unknown size.

• Added lfs_fs_grow which allows you to change the size of an existing filesystem, thanks to @kaetemi (#872)

  The main use case is for changing the size of a partition as storage requirements change.

  Note: littlefs does not support shrinking a filesystem, and this is unlikely to change (it's a hard problem).

• Added lfs_fs_gc which allows you to manually run the block allocator, thanks to @opilat (#875)

  Note: littlefs currently does not persist any found free blocks to disk, though this may change in the future.

  This limits the current value of lfs_fs_gc, but it may still be useful for moving the expensive block scan out of performance sensitive code paths.

Changes

f91c5bd Bumped minor version to v2.8 6b33ee5 Renamed lfs_fs_findfreeblocks -> lfs_fs_gc, tweaked documentation 63e4408 Extended alloc tests to test some properties of lfs_fs_findfreeblocks dbe4598 Added API boilerplate for lfs_fs_findfreeblocks and consistent style d85a0fe Move lookahead buffer offset at the first free block if such block doesn't exist move it for whole lookahead size. b637379 Update lfs_find_free_blocks to match the latest changes. 23505fa Added lfs_fs_grow for growing the filesystem to a different block_count 2c222af Tweaked lfs_fsinfo block_size/block_count fields 127d84b Added a couple mixed/unknown block_count tests 027331b Adopted erase_size/erase_count config in test block-devices/runners 9c23329 Revert of refactor lfs_scan_* out of lfs_format 23089d5 remove previous block_count detection from lfs_format d6098bd Add block_count and block_size to fsinfo d6c0c6a linting 5caa83f forgot to unmount lfs in test; leaking memory 7521e0a fix newly introduced missing cleanup when an invalid superblock is found. 2ebfec7 test for failure when interpretting block count when formatting without superblock 3d0bcf4 Add test_superblocks_mount_unknown_block_count 6de3fc6 fix corruption check df238eb Add a unit test; currently hanging on final permutation. be68122 introduce lfs->block_count. If cfg->block_count is 0, autopopulate from superblock 6dae703 remove redundant superblock check 7328527 refactor lfs_scan_for_state_updates and lfs_scan_for_superblock out of lfs_format

5a834b6 initlize struct lfs_diskoff disk = {0} so we don't use it uninitlized in first run d775b46 Fixed integer conversion warning from Code Composer Studio 96fb8be Fixed release script breaking if there is no previous version

This is a small release to add some features that help with the previous on-disk minor-version bump, per feedback from @rojer, @TjoBitDk, and others.

What's new?

• Added lfs_fs_stat as an analog for the POSIX stavfs (#838)

  This now provides access to the on-disk format-time configuration:

  • disk_version - on-disk version
  • name_max - configurable name limit
  • file_max - configurable name limit
  • attr_max - configurable name limit

  And may be extended in the future. Notably, the current block avail/usage is missing due to performance concerns, but can be retrieved with an explicit call to lfs_fs_size.

• Added support for writing previous on-disk minor-versions (#839)

  This feature is behind a compile-time flag, so in order to write lfs2.0 you need to compile with LFS_MULTIVERSION defined. LFS_MULTIVERSION enables an additional config option, disk_version, which currently supports three options:

  • disk_version=0 => write the most recent littlefs version
  • disk_version=0x00020001 => write lfs2.1
  • disk_version=0x00020000 => write lfs2.0 (no FCRC)

Changes

a942cdb Bumped minor version to v2.7 79cc75d Added LFS_MULTIVERSION and testing of lfs2.0 to CI eb9af7a Added LFS_MULTIVERSION, made lfs2.0 support a compile-time option b72c96d Added support for writing on-disk version lfs2.0 265692e Removed fsinfo.block_usage for now 08a132e Added a link to ChaN's FatFS implementation c5fb3f1 Changed fsinfo.minor_version -> fsinfo.disk_version 8610f7c Increased context on failures for Valgrind in CI a51be18 Removed previous-version lfsp_fs_stat checks in test_compat a7ccc1d Promoted lfs_gstate_needssuperblock to be available in readonly builds fdee127 Removed use of LFS_VERSION in test_compat 87bbf1d Added lfs_fs_stat for access to filesystem status/configuration

8a4ee65 Removed unnecessary sys/types.h include 6fda813 Fix build for AVR f2bc6a8 Reclassify .toml files as .c files on GitHub

Note!

This release bumps the on-disk minor version of littlefs from lfs2.0 -> lfs2.1.

This change is backwards-compatible, but after the first write with the new version, the image on disk will no longer be mountable by older versions of littlefs.

Read-only operations will not change the on-disk version, and the on-disk version can be explicitly bumped with lfs_fs_mkconsistent if desired.

This is the first on-disk minor version bump for littlefs, and is untested waters, so do open a GitHub issue if this creates any problems.

What's new?

• FCRCs - forward-looking erase-state CRCs (#497)

  Found by @pjsg, littlefs was relying on a bad assumption about the order/atomicity of prog operations to detect incomplete metadata commits. In the case of power-failure, this could lead to littlefs progging the same location multiple times without an erase, leading to best case fixable corruption, worst case significantly reduced data retention.

  The solution implemented here is to store a checksum of the next erased prog-size in every commit. This way any incomplete commits that modify the erased state can be detected.

  In theory this could be implemented in a forward-compatible way, but littlefs currently treats all CRC tags as commit-ending CRC tags, so only updated versions of littlefs will be able to read metadata blocks containing FCRCs. This is the reason for the on-disk minor version bump. On the bright-side, future CRC-related extensions will be forward-compatible.

• A complete overhaul of the testing/bench framework (#752)

  This does not affect users, but involves a long list of needed changes to make it easier to develop/bench littlefs:

  • Tests/benches now run in parallel, and much faster thanks to shared processes.
  • Tests are more reproducible, and narrow bugs can be reproduced locally repeatedly without leaving GDB.
  • More options for power-loss testing. Exhaustive testing of all 1 and 2-deep powerlosses have been added to CI.
  • More scripts related to perf measurements, plotting, tracing, etc.
  • A make help rule to help use/document the Makefile, which is sort of the entry point for littlefs development.
  • Added a CI rule to have @geky-bot comment on each PR with CI measurements (examples here). We'll see if this is an improvement over the GitHub statuses.

  The increased power-loss coverage has already found a couple of difficult to hit bugs (eba5553 and 0b11ce0), and the bench runner will be useful for evaluating littlefs's performance moving forward.

• Added a couple more asserts to help common implementation issues (#801, #802)

• Adopted Brent's algorithm for cycle detection (#809)

  This should significantly speed up cycle detection in the case the metadata linked-list contains a cycle.

• Added lfs_fs_mkconsistent as proposed by @kasper0 (#812)

  lfs_fs_mkconsistent allows users to front-load the internal consistency checks littlefs needs to run before it can write to the filesystem.

  Conveniently this also allows a way to force littlefs to write the new on-disk minor version if that is desired.

• A number of bug fixes, thanks to @ghost, @ajaybhargav, @colin-foster-in-advantage, @rvanschoren, @hgspbs, @sosthene-nitrokey, @kasper0, @Ldd309, and others (#800, #805, #811):

  • #800 - Fixed issues with truncate around block-aligned boundaries
  • #805 - Fixed issue where seeking to end-of-directory return LFS_ERR_INVAL
  • #811 - Fixed issue where lfs_fs_deorphan may run more than needed
  • eba5553 - Fixed issue where half-orphans can hide full orphans in nested power-loss situations
  • 0b11ce0 - Fixed incorrect calculation of extra space needed in mdir blocks
  • ba1c764 - Fixed issue where deorphan could get stuck circling between two half-orpans in power-loss situations

• And a number of links to useful community projects related to littlefs, thanks to @tniessen, @yomimono, @elpiel (#572, #718, #807):

  • littlefs-disk-img-viewer
  • chamelon
  • littlefs2-rust

  If you have a littlefs-related project or a project that uses littlefs, consider opening a PR. Eventually my plan is to move these to a user-contributable wiki.

Changes

259535e Added lfs_fs_mkconsistent 94d9e09 Fixed issue where lfs_fs_deorphan may run more than needed 363a8b5 Tweaked wording of littlefs2-rust link in README.md e43d381 chore: add littlefs2 crate to README 3e25dfc Added FCRC tags and an explanation of how FCRCs work to SPEC.md 9e28c75 Bumped minor version to v2.6 and on-disk minor version to lfs2.1 4c93600 Added ability to bump on-disk minor version ca0da3d Added compatibility testing on pull-request to GitHub test action 116332d Added tests for forwards and backwards disk compatibility f0cc1db Tweaked changeprefix.py to not rename dir component in paths bf045dd Tweaked link to littlefs-disk-img-viewer to go to github repo b33a5b3 Fixed issue where seeking to end-of-directory return LFS_ERR_INVAL 384a498 Extend dir seek tests to include seeking to end of directory b0a4a44 Added explicit assert for minimum block size of 128 bytes aae897f Added an assert for truthy-preserving bool conversions e57402c Added ability to revert to inline file in lfs_file_truncate 6dc18c3 Fixed block-boundary truncate issue d5dc487 Expanded truncate tests to test more corner cases 24795e6 Add missing iterations in tests 7b151e1 Add test scenario for truncating to a block size ba1c764 Fixed issue where deorphan could get stuck circling between two half-orphans d1b254d Reverted removal of 1-bit counter threaded through tags 2f26966 Continued implementation of forward-crcs, adopted new test runners b4091c6 Switched to separate-tag encoding of forward-looking CRCs 91ad673 Cleaned up a few additional commit corner cases 52dd830 Initial implementation of forward-looking erase-state CRCs 1278ec1 Adopted Brent's algorithm for cycle detection c2147c4 Added --gdb-pl to test.py for breaking on specific powerlosses 801cf27 Tweaked/fixed a number of small runner things after a bit of use 1f37eb5 Adopted --subplot* in plot.py cfd4e60 Added --subplot* to plotmpl.py 2d2dd8b Added plotmpl.py --github flag to match the website's foreground/background b0382fa Added BENCH/TEST_PRNG, replacing other ad-hoc sources of randomness d8e7ffb Changed lfs_emubd_get* -> lfs_emubd_* cda2f6f Changed test_runner to run with -Pnone,linear by default 9b687dd Added make benchmarks/testmarks rules c4b3e9d A couple of script changes after CI integration 9990342 Fixed Clang testing in CI, removed override vars in Makefile 4a20934 Fixed bench workflow + changeprefix issue in prefix releases a659c02 Added a bot-generated PR-comment with a simple status table 397aa27 Removed unnecessarily heavy RAM usage from logs in bench/test.py 65923cd Adopted script changes in GitHub Actions 387cf6f Fixed a couple corner cases in scripts when fields are empty 0b11ce0 Fixed incorrect calculation of extra space needed in mdir blocks eba5553 Fixed hidden orphans by separating deorphan search into two passes f89d758 Fixed test out-of-space issues with powerloss testing 6c18b4d Added a simple help rule to the Makefile f734941 Changed default build target lfs.a -> liblfs.a bcc88f5 A couple Makefile-related tweaks e35e078 Renamed prefix.py -> changeprefix.py and updated to use argparse 1a07c2c A number of small script fixes/tweaks from usage 6fce9e5 Changed plotmpl.py/plot.py to not treat missing values as discontinuities 559e174 Added plotmpl.py for creating svg/png plots with matplotlib b2a2cc9 Added teepipe.py and watch.py 3a33c37 Added perfbd.py and block device performance sampling in bench-runner 29cbafe Renamed coverage.py -> cov.py df283ae Added recursive results to perf.py 490e1c4 Added perf.py a wrapper around Linux's perf tool for perf sampling ca66993 Tweaked scripts to share more code, added coverage calls/hits 296c5af Renamed bench_read/prog/erased -> bench_readed/proged/erased 274222b Added some automatic sizing for field-names in scripts/runners a2fb708 Added stddev/gmean/gstddev to summary.py 9507e62 Several tweaks to script flags 42d889e Reworked/simplified tracebd.py a bit fb58148 Consistent handling of by/field arguments for plot.py and summary.py 7591d9c Added plot.py for in-terminal plotting 9a0e3be Added a quick trie to avoid running redundant test/bench permutations 4fe0738 Added bench.py and bench_runner.c for benchmarking 20ec0be Cleaned up a number of small tweaks in the scripts 11d6d12 Dropped namespacing of test cases 1fcd82d Made test.py output parsable by summary.py acdea18 Made summary.py more powerful, dropped -m from size scripts 23fba40 Added option for updating a CSV file with test results 03c1a4e Added permutations and ranges to test defines bfbe44e Dropped permutation number for full leb16-encoded defines 5a2ff17 Changed test identifier separator # -> : c7f7094 Several tweaks to test.py and test runner a208d84 Reworked test defines a bit to use one common array layout 91200e6 Added tracebd.py, a script for rendering block device operations c9a6e3a Added tailpipe.py and improved redirecting test trace/log output over fifos 5279fc6 Implemented exhaustive testing of n nested powerlosses 552336e Added optional read/prog/erase delays to testbd 3f4f859 Readded support for mirror writes to a file in testbd 4689678 Added --color to test.py, fixed some terminal-clobbering issues 61455b6 Added back heuristic-based power-loss testing 01b11da Added a simple test that the block device works a368d3a Moved emulation of erase values up into lfs_testbd b08463f Reworked scripts/pretty_asserts.py a bit 92eee8e Removed some prefixes from Makefile variables where not necessary d9333ec Add "chamelon" to the related projects section. 46cc6d4 Added support for annotated source in coverage.py 5b0a6d4 Reworked scripts to move field details into classes 4a7e94f Reimplemented coverage.py, using only gcov and with line+branch coverage 2b11f2b Tweaked generation of .cgi files, error code for recursion in stack.py 1616115 Fix test.py hang on ctrl-C, cleanup TODOs 4a42326 Moved test suites into custom linker section 0781f50 Ported tests to new framework d679fbb In ./scripts/test.py, readded external commands, tweaked subprocesses 5a572ce Reworked how test defines are implemented to support recursion be0e6ad More progress toward test-runner feature parity 4962829 Continued progress toward feature parity with new test-runner 5ee4b05 Misc test-runner improvements 5812d2b Reworked how multi-layered defines work in the test-runner 6443693 Putting together rewritten test.py script 92a600a Added trace and persist flags to test_runner 9281ce2 More test_runner progress 4b0aa62 Some more minor improvements to the test_runner d683f1c Reintroduced test-defines into the new test_runner 56a9903 Created new test_runner.c and test_.py 3ae87f4 Add littlefs-disk-img-viewer to README