PackSquash Versions Save

Additions

• As requested in #44, basic support for Minecraft 1.17 include shaders was added. Please note that PackSquash is, for now, more restrictive than Minecraft when parsing these files, imposing that they must be a syntactically valid translation unit by themselves. In other words, include shaders that would only be valid within an external declaration of another shader, such as those containing only variable assignment expressions, are not accepted. We are looking forward to lifting this restriction in the future. Thanks to Ancientkingg#0420 over Discord for making me notice this quirk.
• Language files used in Minecraft 1.12.2 and older versions are now fully supported. Thanks to a Discord user whose nickname I can't find right now for bringing this to my attention.
• Added support for the resource pack assets used by the Custom Resources feature of the Minecraft Transit Railway mod, major version 3. Thank you @Kenny-Hui for getting in touch over Discord to suggest this improvement and for your keenness to answer technical questions about the mod. The tracking issue for this feature is #42.
• Custom assets can now be defined in options files by setting the file-specific option force_include to true. This option is honored by PackSquash for files that would otherwise be unrecognized and skipped from the generated ZIP file, and it can be used to include files that are not read by the game in the generated pack, such as credits or license information. Please get in touch if you feel the urge to use this feature to compensate for PackSquash lacking proper support for asset files used by the game, though! Thank you @MinecraftAdmin for suggesting this improvement on issue #40.

Compression improvements

• PackSquash now only processes files whose path in the pack matches a hardcoded list of path patterns. Previously, any file that had a recognized extension, such as .json, was optimized and included in the generated ZIP file, even if there was no way for Minecraft to read it. This improvement allows PackSquash to recognize different asset types much more precisely, enabling more specific and advanced optimizations. It also allowed the PackSquash codebase to be greatly refactored and simplified. However, this is a breaking change for users relying on PackSquash to optimize files that are not used by Minecraft: Java Edition, or only used by unsupported mods that do not follow the same asset location scheme as vanilla Minecraft.
• Merged @sya-ri's PR #45, which ensures that Blockbench cube groups data is removed from JSON model files when debloating is enabled. Thank you very much for the PR!
• The JSON model debloat code now removes keys conventionally used by some authors to hold comment text, but completely ignored by the game.
• The Ogg Vorbis encoder used is now told to disregard ease of seeking of the files it generates, as Minecraft does not need to rewind or fast forward audio. Consequently, longer sounds are now more likely to be transcoded to files with less muxing overhead, slightly reducing their size with no changes to audio quality.
• When keeping the color_quantization_target option value to the new default of auto, now PackSquash won't use the color-quantized representation of a texture if it does not yield space savings. The previous behavior of always quantizing to 256 colors can be restored by explicitly setting color_quantization_target to the previous default value. In addition, a quick, never size-increasing first optimization pass is now done on PNG files, and unless expressly requesting color quantization to be done, its results will be used if PackSquash can't figure out how to make the input PNG smaller. Thanks Mergu#0001 for sharing a previously problematic texture for analysis.
• The dithering effect introduced while performing color quantization on textures is now slightly less intense and can be configured via the color_quantization_dithering_level option. This is meant to balance visual quality with size savings better.
• Several tweaks were made to the Zopfli iteration count calculation code for PNG files to make it behave more as originally envisioned. PackSquash should now spend a bit more time optimizing smaller PNG images as it did on v0.2.1 while still hurrying up for bigger images as introduced on v0.3.0. The default value of the image_data_compression_iterations option was increased to 5 to reflect this change and allow reducing it even more.
• Command function files in data packs are now minified by PackSquash, removing comments and empty lines. Thank you @sya-ri for submitting this improvement on PR #68!
• The sampling_frequency option value can now only lower the sampling frequency of audio files, not increase it. The rationale for the new behavior is that upsampling sounds of an initially lower sampling frequency increases file sizes due to the need to store data about the additional samples, but no audio information is added. However, if the lowest sampling frequency chosen for an audio file is not supported by the encoder used by PackSquash, it will error out. This could also happen before when using certain values for the sampling_frequency option.
• Introduced the internal asset type mask abstraction, used throughout the source code to know what asset types are appropriate for the targeted Minecraft versions. Thanks to this abstraction, PackSquash can skip files only used in older Minecraft versions when the pack targets newer versions according to its pack.mcmeta file, for example. The automatic_asset_types_mask_detection option was added to control how this asset type mask feature operates. Setting the validate_pack_metadata_file option to false will now only work if automatic_asset_types_mask_detection is set to false, too, as this feature needs to validate the pack metadata file too.

Performance improvements

• The thread synchronization primitives used internally were replaced with more efficient implementations, yielding modest total pack processing time savings. Thanks to @clrxbl and jilchu#4474 for your spot-on comments about PackSquash's performance which lead me to make this and other performance-related changes.
• JSON debloating is now limited to files that contain a known-debloatable asset type, and any debloating done is specific to the asset type. In addition to improving performance by doing less unnecessary work, this behavior is more accurate than the previous one of applying every debloating technique to every JSON file. End-users can know whether a JSON file was debloated or not thanks to a different optimization strategy message, which will now have the "and debloated" suffix when debloating.
• Output ZIP file operations now use a single lock to synchronize access to their mutable state instead of several locks acquired in succession, slightly reducing the synchronization cost of these operations.
• The official Linux and Windows PackSquash executables for x64 CPUs now target the second microarchitecture level of x64, allowing PackSquash to use more efficient instructions found on most, but not all, x64 processors. Any full-featured desktop, laptop, workstation, or server CPU manufactured on or after 2009-2010 should be compatible.
• Other minor micro-optimizations.

Pack file validation improvements

• PackSquash now sanity-checks that JSON files belonging to a known asset type (Minecraft model, etc.) are JSON objects. Minecraft and some JSON standards expect JSON data to be contained within a JSON object.
• Added an always_allow_json_comments JSON-specific file option to control whether PackSquash allows comments in the matched files or not. The default value is true, which allows and removes comments from JSON files. When this option is set to false, comments will only be accepted in files whose usual extension ends with an extra c letter, such as .jsonc, .mcmetac or .jpmc.

Fixes

• Fixed issue #43 about shaders containing functions with parameters or a single statement being broken by PackSquash when minifying them. Thanks to @tomlister and @Aiamded for reporting this.
• Shaders that contain a preprocessor directive following an external declaration, such as a variable declaration, are no longer corrupted by PackSquash minifications. Thank you Esron#1571 for reporting this issue over Discord.
• Fixed sounds in resource packs not playing when the zip_spec_conformance_level and percentage_of_zip_structures_tuned_for_obfuscation_discretion options are set to disregard and 0, respectively. Thank you @Aiamded for reporting this issue.
• Resolved issues #37 and #39 by adding the restrictive_banner_layer_texture_format_check and bad_entity_eye_layer_texture_transparency_blending quirks, respectively, which PackSquash automatically applies when necessary by default. Thanks to ! " Marco Pollo#1898 and @Aiamded, respectively, for reporting these problems.
• When PackSquash ends up normalizing two files to the same path, which can happen when changing, for example, the extension of specially named .jsonc files to .json, an error will now happen again, as it did in v0.2.1. v0.3.0 introduced a regression that made it possible to leverage this normalization to store two files with the same name on the generated ZIPs, negatively impacting their sizes for no good reason.

User experience improvements

• PackSquash progress messages are now shown with color and emojis on supported platforms, making them prettier and facilitating the visual location of message types. This work has been done by @sya-ri on PR #46. Thank you!
• A title is now shown on the interactive console or terminal attached to PackSquash, showing the application name and the progress of the optimization process.
• When an attempt is made to reuse a previously generated ZIP file, but that is not possible due to system identifier changes or other reasons, PackSquash will now automatically ignore that previous ZIP file after showing a warning instead of exiting with an error. This change is a concession to the fact that external programs or scripts are likely to handle this condition by just retrying without the previous ZIP file: doing otherwise is not practical in most scenarios. Also, non-savvy end users might be confused by the former behavior. Those who are savvy enough can still interrupt the optimization after reading that warning and apply a fix to the underlying cause, should they choose to do so.
• PackSquash now promptly shows a unique, user-friendly error message if the value of the output_file_path option points to an existing directory, and thus it cannot be a normal file. Experience has shown that this is a common mistake, and Windows reported a generic "access denied" error message after the whole pack was optimized, confusing users.
• The help texts shown when reading options from the standard input and after a pack processing error happens were tweaked, with the intent of making them easier to understand and better highlight helpful solutions or alternatives.
• The error message that shows when trying to reuse an empty or too small previously generated ZIP file was changed to a less generic, more user-friendly one.
• The error message shown when validating a pack.mcmeta file that does not contain a valid pack_format key now correctly points to that key instead of pack_format_version.
• Some common error messages shown for PNG files were changed to be more actionable and user-friendly.
• PackSquash now shows both the count of files it found on the pack directory and the count of pack files that were actually stored on the generated ZIP file. This may be handy for troubleshooting purposes and quickly knowing whether files were skipped.

Distribution changes

• macOS executables now work with the upstream GStreamer libraries instead of Homebrew ones. The migration was necessary because the latest Homebrew packages have broken plugin resolution, and installation instructions were updated accordingly. As a side bonus, now the pitch-shifting feature also works on macOS.
• The PackSquash Linux CI build workflows now generate a PackSquash AppImage. AppImages are universal Linux packages that integrate relatively well with most versions of most major distros. It even is possible to run the generated AppImages on musl Linux environments, such as Alpine, but it's necessary to extract them beforehand on a glibc environment in this case. AppImages are now the recommended way to run PackSquash when more environment-specific packages are not provided or can't be used.
• The PackSquash Linux CI build workflows are now executed in Debian Bullseye containers, which markedly improved build reproducibility and predictability, availability of development packages, and compatibility with a wide range of distros (stable Debian releases tend to ship older, but forward-compatible glibc versions than most other distros). This change should also make it easier to replicate the PackSquash build environment for private builds. However, executables built in Debian Bullseye are not compatible with Ubuntu 18.04 or any distro that uses a glibc version older than 2.31. The recommended migration path for cases where upgrading is not feasible is to use the PackSquash AppImage.
• The distribution of PackSquash executables for Linux is now deprecated. AppImages are a better alternative to them in all regards, as they are self-contained and more compatible across distros. Future releases may stop distributing PackSquash executables for Linux. For now, they are kept to help their users ease and plan for the transition to AppImages or distribution-specific packages.

Codebase changes

• Unit tests for the official Linux AArch64/ARM64 builds are now run via QEMU AArch64 user-mode emulation, increasing the confidence that these builds work as expected for end-users and paving the way towards recognizing them as fully-supported ports.
• An automated Criterion.rs microbenchmark suite was added to the project. The benchmark suite runs automatically on each commit thanks to a new GitHub Actions job, adding a comment and updating a performance evolution graph, enabling continuous performance monitoring and improvement. Future changes may leverage these benchmarks to establish baselines, assessing their performance impact with hard data. @victorlf4 contributed a Python script to sample packs in smaller ones and shared some mathematical optimization ideas. jilchu#4474 and @Aiamded kindly authorized the inclusion of their packs in the benchmarking dataset.
• The Rust source tree was refactored into a Cargo workspace with independent library and CLI packages. This change promotes separation of concerns, as the CLI package uses the library package as a dependency, like external code would, and frees the library package from any UI-related code. It is expected for this change to help achieve a better API design and make it easier to add new UI frontends for PackSquash that may be developed independently from the PackSquash core optimization routines.
• Contributing guidelines were added to the repository. Thank you @sya-ri for suggesting these.
• A code of conduct was added to the repository, documenting the behavior we've already been expecting of the PackSquash contributors community.
• A security policy was added to the repository.
• Updated the codebase to the Rust 2021 edition, bringing along a new dependency resolver that generates potentially smaller executables in some cases.
• The build workflow responsible for generating unstable PackSquash builds is now run at least once a week, helping ensure that PackSquash still builds with newer Rust toolchain versions and that programs consuming the latest unstable build can always download one.
• A rust-toolchain rustup override file was checked-in to the repository, making the nightly Rust toolchain requirement much more transparent for fellow developers and automated tools.
• The static analysis GitHub Actions build workflow tasks are now run in a concurrent, separate job. A step that runs the cargo-deny Cargo plugin was added.
• Some dependencies were patched to avoid several versions of the same transitive dependencies being compiled to the executable. This was discovered thanks to cargo-deny checks. These patches streamlined executable sizes.
• The audio-transcoding Cargo feature was added to choose at compile time whether PackSquash implements audio transcoding functionality using GStreamer. This feature is enabled by default, and all official builds have this feature enabled. Disabling the feature stops the PackSquash executable from linking to GStreamer libraries, which may be helpful for some advanced use cases, but disables parsing of audio file optimization options and the optimizations themselves, turning audio files into assets that are merely copied as-is.
• A logging framework is now used to show status and progress messages, allowing for further expandability. The PACKSQUASH_LOG environment variable can be used to customize what logging messages are shown, whose contents are parsed by the env_logger library, but this is considered for now an implementation detail that may change in the future. @sya-ri did this change as a part of PR #46.

Miscellaneous

• Trailing bytes at the end of PNG files are now considered errors, as they violate the PNG format standard, do not make sense to store, and pose compatibility problems with select programs.
• The ZIP file path handling code was refactored to check the file path length. This saves repeated length checks over the code, making it more efficient and maintainable.
• Several dependency updates, bringing the latest upstream fixes and improvements.
• Updated GitHub Actions build workflow runners to their latest versions.
• Other minor refactors to PackSquash, its documentation, and the GitHub Actions build workflow.
• JetBrains now provides Open Source Development licenses of their development tools for core PackSquash contributors.

Options files changes

• The following options are new (i.e. they were introduced in v0.3.1 and do not have an equivalent in v0.3.0):
  • automatic_asset_types_mask_detection
  • color_quantization_dithering_level
  • always_allow_json_comments
  • minify_legacy_language
  • strip_legacy_language_bom
  • minify_command_function
  • force_include
• The following values were added to the color_quantization_target option:
  • auto (new default value)
• The following values were added to the work_around_minecraft_quirks option:
  • restrictive_banner_layer_texture_format_check
  • bad_entity_eye_layer_texture_transparency_blending
• The following values were added to the allow_mods option:
  • Minecraft Transit Railway 3
• The default value of image_data_compression_iterations was changed to 5 from 3.

Additions

• Text files used by Minecraft that contain the End Poem, credits and main screen splashes are now supported.
• Initial support for data packs was added, by handling .nbt and .mcfunction files. Currently, these data pack specific files are just copied to the generated ZIP file, but improved file-specific optimization techniques will be introduced in the future.
• JSON files (.json, .mcmeta and, if OptiFine is allowed, .jem and .jpm) can now contain Unix shell style line comments (# line comment), C style block comments (/* block comment */), and C style line comments (// line comment). Comments are ignored and removed from the optimized JSON that is stored in the ZIP file. JSON with comments files (.jsonc) are now supported, and their extension is canonicalized to .json in the generated ZIP file, as Minecraft expects.
• Pack file names can now at least contain every character in the Unicode Basic Multilingual Plane. This is a substantial improvement from the previous limitation of only ASCII characters.
• The pack.mcmeta file is now validated for egregious errors by default, like missing pack_format or description keys. This validation can be disabled if necessary.
• An APT repository was created, to make it easier to install PackSquash and keep it updated on Debian-like Linux distros on the officially supported architectures. This is now the recommended method to install PackSquash on distros that can use this repository, like Debian and Ubuntu.

Compression improvements

• The PackSquash ZIP compressor was greatly revamped to allow for identical file deduplication and a greater range of ZIP specification conformance levels, so you can now choose the right balance between optimization and interoperability for your needs in a more precise way.
• Key-value pairs in JSON files that are known to be ignored by Minecraft, but usually present anyway, are now removed. This behavior can be disabled.
• The number of Zopfli compression iterations applied to pack files to store them in the generated ZIP file is now variable and configurable. This is useful to customize the desired tradeoff between compression and performance. The precise number of iterations done to a file depends on its size, with smaller files being compressed more, and larger files being compressed less, so performance is more predictable and consistent. As packs used with PackSquash tend to contain lots of small files but only a handful of big ones that are not already compressed, this new behavior can yield more space savings.
• PNG files can now be quantized to 2, 4 and 16 colors, in addition to the previously hardcoded, but still default, 256 colors.
• Improved shader minification. The exact savings depend on the input file, with bigger and more intrincate files being subject to more savings, but some test cases have shown a more than 10% optimized size decrease, compared to the previous shader minification technique.
• The generated ZIP files can now make use of ZIP64 extensions when needed to overcome certain limits of the original ZIP file format, like a maximum of 65535 files, or a total of less than 4 GiB of compressed data. Note that this ZIP64 support does not improve the per-file 4 GiB limit, because such big files do not make sense to use with Minecraft.

Performance improvements

• ZIP files that PackSquash has generated in previous runs can now be automatically used to vastly speed up next runs, as unchanged files can just be copied from that ZIP file instead of being processed again. PackSquash relies on the file modification timestamps provided by the filesystem to quickly deduce if a file may have been changed.
• Gigantic images (> 8192x8192) are now rejected by default, because they usually make little sense to use with Minecraft, unless they are atlases or animated textures. Limiting this ensures that PackSquash does not take too long processing them and helps authoring resource packs that work well accross a wide range of maximum GPU texture sizes.
• Images that have a lot of pixels are now optimized much faster, thanks to an adaptative compression strategy that calculates an appropriate number of Zopfli compression iterations beforehand to meet acceptable performance targets. This behavior can be tuned to be even faster in exchange for worse compression, or prioritize compression at the expense or performance.
• Buffering is now used by default in a way that is better adapted to the device PackSquash runs on and can be configured by the user. This substantially improves performance when dealing with big (> 64 MiB) packs if there is plenty of available memory, while also improving stability in environments with low available memory. When a buffer gets rolled over to disk because the available memory is exhausted, now it will be replaced by a much smaller buffer, so performance does not drop as much as it did before.
• PackSquash is now designed around a multithreaded asynchronous programming model, as opposed to the previously used multithreaded synchronous programming model. This new model leverages cooperative task scheduling to improve thread utilization. Some internal tests showed that the total execution time was reduced by ≈5%, but this figure may change depending on the contents of your pack and the device you run PackSquash on.
• Lots of micro-optimizations in regards to both memory and CPU usage.

Extraction protection improvements

• Now PackSquash uses much more sophisticated extraction protection techniques, with additional configuration knobs that allow tuning the generated ZIP file for increased compressibility or obfuscation. These extraction protection techniques were possible to implement thanks to original research on Minecraft internals, in addition to suggestions and ideas from some Discord users.

Fixes

• Allowing the OptiFine mod now properly deals with the Custom Entity Models feature, which adds .jem and .jpm files to resource packs. Previously these files were treated as unknown junk files and skipped.
• Fixed issue #12, which made grayscale-looking color images look wrong on older versions of Minecraft, by working around the underlying Minecraft quirk.
• Input PNG files with embedded gamma information are now corrected to a display gamma value of 2.2, which closely matches that of the sRGB color space, because this is the gamma that Minecraft and most other end-user applications and devices assume and use. Previously, input images with an unconventional gamma would have that gamma information stripped without their colors being corrected, which led to wrong-looking results down the line.
• Fixed issue #13 as much as possible.
• JSON, shader and OptiFine-added text files with a UTF-8 BOM now work with PackSquash. The BOM is routinely added to UTF-8 text files by some editors, like Windows Notepad, but it serves no purpose within a pack, so it is now ignored and removed.
• Audio files are no longer downmixed to mono by default, because the different 3D effects that downmixing may imply can be inappropriate, and doing so did not yield very significant space savings anyway, due to Ogg Vorbis joint encoding.
• Shader files with Windows (CR + LF) line endings now work fine with PackSquash.
• Shaders that contain the #elif preprocessor directive can now be parsed and optimized by PackSquash. The non-standard #elseif preprocessor directive is no longer accepted.
• Fixed issue #34 about the #moj_import preprocessor directive added in Minecraft 1.17 not being supported in shaders.
• Some GLSL statements are no longer broken by PackSquash optimizations.
• The OxiPNG library is now told to stop trying further optimizations on images if the process has been running for more than ten minutes. This deadline is just a best-effort.

Stability improvements

• PackSquash' software architecture was reworked to support dealing with files larger than the available memory. Nevertheless, the usefulness of this change is limited in practice by libraries that require the entire file to be addressable in main memory to be processed. Currently, only audio and copied files can be processed without storing their entire contents in memory.
• Now PackSquash aborts its execution if an error occurs while processing the pack, instead of continuing and risking generating corrupt or incomplete ZIP files. This behavior also allows using PackSquash as a pack validator more efficiently.
• A configurable limit of simultaneously open files was introduced. This avoids errors caused by the open file limit imposed by the operating system being reached, which can happen in big packs if lots of threads are used.

Codebase changes

• The entire codebase was refactored to be much more modular, testable, and maintainable. In particular, the application and logic code were separated in two binary and library crates, respectively, promoting separation of concerns and paving the way for a future API.
• Automated unit tests were added for the most critical components of PackSquash. This increases the confidence that those components work okay, and continue to work fine after changes. These tests are run automatically for each build.
• Improved build reproducibility by checking in to the repository the Cargo.lock file.
• Improved the GitHub Actions workflow, fixing semantic version calculations, build artifacts caching, and other changes.
• Visual Studio Code configuration files were checked in to the repository. This should help new contributors configure their development environment more easily.
• An .editorconfig file was added to improve code readability in the GitHub website.
• Updated libraries and toolchain components for the latest fixes and improvements.

Miscellaneous

• JSON files can now be prettified instead of minified before storing them in the generated ZIP file in a per-file basis. This may be useful for packs under a permissive license that wants to encourage contributions from its users. Similarly, minification of shader and .properties files can now be disabled.
• Opus audio files are now supported as input files in resource packs. They are transcoded to Ogg so that Minecraft can read them.
• Several tidy-ups and improvements to the text that PackSquash outputs. The error message shown when the output path file was a folder was improved to be much more user-friendly.
• Detailed version, author and license information is now shown on startup. More detailed information can be seen with the -v command line switch.
• The hardcoded list of system files that are ignored when ignore_system_and_hidden_files is set to true was expanded and improved.
• Progress messages are now printed to the standard error stream, following the trend and conventions established by other command line applications.
• Official builds for the Linux AArch64/ARM64 platform are now available. Even though these builds should work okay, support for this platform is limited to a best-effort, because PackSquash has not actually been run on this platform on the device of a repository maintainer.
• Revamped the panic handling code to output prettier and more user-friendly error and backtrace information, without compromising its technical usefulness.
• Now PackSquash shows, under certain circumstances, URLs pointing to contextually relevant documentation.
• PackSquash now exits with the status code 128 if any error happens while processing a pack. If the error was caused by a ZIP file that PackSquash attempts to reuse but it is not able to, the returned status code is 129.
• Information for sponsoring PackSquash developers was added to the README.md file. A FUNDING.yml file was also added so that GitHub displays a relevant "Sponsor" button in its web interface.

Options files changes

• Settings files have been renamed to options files.
• The following options are new (i.e. they were introduced in v0.3.0 and do not have an exact equivalent in v0.2.1):
  • zip_compression_iterations
  • work_around_minecraft_quirks
  • size_increasing_zip_obfuscation
  • percentage_of_zip_structures_tuned_for_obfuscation_discretion
  • never_store_squash_times
  • spooling_buffers_size
  • open_files_limit
  • zip_spec_conformance_level
  • validate_pack_metadata_file
  • automatic_minecraft_quirks_detection
  • color_quantization_target
  • image_data_compression_iterations
  • skip_alpha_optimizations
  • minify_json
  • delete_bloat_keys
  • maximum_width_and_height
  • minify_shader
  • minify_properties
• The following options were superseded:
  • strict_zip_spec_compliance, by zip_spec_conformance_level
  • quantize_image, by color_quantization_target
• The following options were renamed, with no change in functionality or values they accept:
  • resource_pack_directory → pack_directory
  • compress_already_compressed_files → recompress_compressed_files
  • allowed_mods → allow_mods

Additions

• Text files used by Minecraft that contain the End Poem, credits and main screen splashes are now supported.
• The set of Minecraft quirks that PackSquash needs to work around is now automatically detected by default. It is still possible to manually specify the set of quirks via the options file, though.
• The pack.mcmeta file is now validated for egregious errors by default, like missing pack_format or description keys. This validation can be disabled if necessary.
• An APT repository was created, to make it easier to install PackSquash and keep it updated on Debian-like Linux distros on the officially supported architectures. This is now the recommended method to install PackSquash on distros that can use this repository, like Debian and Ubuntu.
• Introduced a stick parity bit in the Squash Time storage format, to probabilistically detect whether a previously generated ZIP file can indeed be reused. For realistic packs, the detection works reliably and avoids most pitfalls related to using this feature effectively.

Compression improvements

• Improved shader minification. The exact savings depend on the input file, with bigger and more intrincate files being subject to more savings, but some test cases have shown a more than 10% optimized size decrease, compared to the previous shader minification technique.

Performance improvements

• Images that have a lot of pixels are now optimized much faster, thanks to an adaptative compression strategy that calculates an appropriate number of Zopfli compression iterations beforehand to meet acceptable performance targets. This behavior can be tuned to be even faster in exchange for worse compression, or prioritize compression at the expense or performance.
• Minor micro-optimizations in regards to memory usage.

Fixes

• ZIP files generated when the zip_spec_conformance_level option is set to disregard no longer fail to load sometimes with older Minecraft versions, if PackSquash is instructed to work around a newly-added quirk. (Thanks Dvalin#5304 for reporting this over Discord)
• Shaders that contain the #elif preprocessor directive can now be parsed and optimized by PackSquash. The non-standard #elseif preprocessor directive is no longer accepted.
• Fixed issue #34 about the #moj_import preprocessor directive added in Minecraft 1.17 not being supported in shaders.
• Some GLSL statements are no longer broken by PackSquash optimizations.
• PNG file optimization errors caused by exceeding the configured image width and height limits now show a proper error message, instead of a totally unrelated one like "invalid bit depth".
• Pack directory relative paths that started with . and .. are now properly dealt with, like they were on v0.2.1.
• Fixed the last Windows service pack install date not being considered a volatile system identifier as it was intended.
• PackSquash for Linux now works on distros that ship an slightly older version of the GNU C library, like Debian Buster, as v0.2.1 did.
• File sizes are no longer used in some cases to decide execution flow, which prevents a class of race conditions from happening in rare circumstances.
• Each PackSquash build no longer has a different application salt for no reason, making the ZIP file reuse feature more effective across builds, as originally intended. Nevertheless, the application salt can still change any time without notice.

Stability improvements

• Pack processing is now interrupted properly even if a panic happens.

Miscellaneous

• Revamped the panic handling code to output prettier and more user-friendly error and backtrace information, without compromising its technical usefulness.
• Several minor dependency updates.
• Removed some development dependencies that were no longer necessary.
• Some strings that PackSquash outputs were tweaked with the intent of making their meaning clearer for all users.
• Now PackSquash shows, under certain circumstances, URLs pointing to contextually relevant PackSquash documentation.
• The .tmp extension was added to the hardcoded list of system files extensions that ignore_system_and_hidden_files uses when set to true.
• PackSquash now exits with the status code 128 if any error happens while processing a pack. If the error was caused by a ZIP file that PackSquash attempts to reuse but it is not able to, the returned status code is 129.

Options files changes

• The default value for maximum_width_and_height was changed to 8192, as a compromise between rejecting too big single textures and allowing PackSquash to work with most animated textures and atlases by default.
• The grayscale_textures_gamma_miscorrection quirk was renamed to grayscale_images_gamma_miscorrection to be more consistent with the names PNG files receive in the documentation.
• Added the java8_zip_parsing quirk.
• The following options were introduced:
  • validate_pack_metadata_file
  • automatic_minecraft_quirks_detection
  • image_data_compression_iterations
  • skip_alpha_optimizations

Additions

• Initial support for data packs was added, by handling .nbt and .mcfunction files. Currently, these data pack specific files are just copied to the generated ZIP file, but improved file-specific optimization techniques will be introduced in the future.
• JSON files (.json, .mcmeta and, if OptiFine is allowed, .jem and .jpm) can now contain Unix shell style line comments (# line comment), C style block comments (/* block comment */), and C style line comments (// line comment). Comments are ignored and removed from the generated ZIP file. JSON with comments files (.jsonc) are now supported, and their extension is canonicalized to .json in the generated ZIP file, as Minecraft expects.
• Pack file names can now at least contain every character in the Unicode Basic Multilingual Plane. This is a substantial improvement from the previous limitation of only ASCII characters.

Compression improvements

• The PackSquash ZIP compressor was greatly revamped to allow for identical file deduplication and a greater range of ZIP specification conformance levels, so you can now choose the right balance between optimization and interoperability for your needs in a more precise way.
• Key-value pairs in JSON files that are known to be ignored by Minecraft, but usually present anyway, are now removed. This behavior can be disabled.
• The number of Zopfli compression iterations applied to pack files to store them in the generated ZIP file is now variable and configurable. This is useful to customize the desired tradeoff between compression and performance. The precise number of iterations done to a file depends on its size, with smaller files being compressed more, and larger files being compressed less, so performance is more predictable and consistent. As packs used with PackSquash tend to contain lots of small files but only a handful of big ones that are not already compressed, this new behavior can yield more space savings.
• PNG files can now be quantized to 2, 4 and 16 colors, in addition to the previously hardcoded, but still default, 256 colors.
• The generated ZIP files can now make use of ZIP64 extensions when needed to overcome certain limits of the original ZIP file format, like a maximum of 65535 files, or a total of less than 4 GiB of compressed data. Note that this ZIP64 support does not improve the per-file 4 GiB limit, because such big files do not make sense to use with Minecraft.

Performance improvements

• ZIP files that PackSquash has generated in previous runs can now be automatically used to vastly speed up next runs, as unchanged files can just be copied from that ZIP file instead of being processed again. PackSquash relies on the file modification timestamps provided by the filesystem to quickly deduce if a file may have been changed.
• Gigantic textures (> 4096x4096) are now rejected by default, because they usually make little sense to use with Minecraft, unless they are atlases. Limiting this ensures that PackSquash does not take too long processing them and helps authoring resource packs that work well accross a wide range of maximum GPU texture sizes.
• Buffering is now used by default in a way that is better adapted to the device PackSquash runs on and can be configured by the user. This substantially improves performance when dealing with big (> 64 MiB) packs if there is plenty of available memory, while also improving stability in environments with low available memory. When a buffer gets rolled over to disk because the available memory is exhausted, now it will be replaced by a much smaller buffer, so performance does not drop as much as it did before.
• PackSquash is now designed around a multithreaded asynchronous programming model, as opposed to the previously used multithreaded synchronous programming model. This new model leverages cooperative task scheduling to improve thread utilization. Some internal tests showed that the total execution time was reduced by ≈5%, but this figure may change depending on the contents of your pack and the device you run PackSquash on.
• Lots of micro-optimizations in regards to both memory and CPU usage.

Extraction protection improvements

• Now PackSquash uses much more sophisticated extraction protection techniques, with additional configuration knobs that allow tuning the generated ZIP file for increased compressibility or obfuscation. These extraction protection techniques were possible to implement thanks to original research on Minecraft internals, in addition to suggestions and ideas from some Discord users.

Fixes

• Allowing the OptiFine mod now properly deals with the Custom Entity Models feature, which adds .jem and .jpm files to resource packs. Previously these files were treated as unknown junk files and skipped.
• Fixed issue #12, which made grayscale-looking color images look wrong on older versions of Minecraft, by working around the underlying Minecraft quirk. This workaround must be explicitly enabled in the options file because it may hurt compression.
• Input PNG files with embedded gamma information are now corrected to a display gamma value of 2.2, which closely matches that of the sRGB color space, because this is the gamma that Minecraft and most other end-user applications and devices assume and use. Previously, input images with an unconventional gamma would have that gamma information stripped without their colors being corrected, which led to wrong-looking results down the line.
• Fixed issue #13 as much as possible.
• JSON, shader and OptiFine-added text files with a UTF-8 BOM now work with PackSquash. The BOM is routinely added to UTF-8 text files by some editors, like Windows Notepad, but it serves no purpose within a pack, so it is now ignored and removed.
• Audio files are no longer downmixed to mono by default, because the different 3D effects that downmixing may imply can be inappropriate, and doing so did not yield very significant space savings anyway, due to Ogg Vorbis joint encoding.
• Shader files with Windows (CR + LF) line endings now work fine with PackSquash.
• The OxiPNG library is now told to stop trying further optimizations on images if the process has been running for more than ten minutes. This deadline is just a best-effort, and does not fully address the fact that some images still take too long to optimize in a way that is not affected by this change.

Stability improvements

• PackSquash' software architecture was reworked to support dealing with files larger than the available memory. Nevertheless, the usefulness of this change is limited in practice by libraries that require the entire file to be copied to memory to be processed. Currently, only audio and copied files can be processed without storing their entire contents in memory.
• Now PackSquash aborts its execution if an error occurs while processing the pack, instead of continuing and risking generating corrupt or incomplete ZIP files. This behavior also allows using PackSquash as a pack validator more efficiently.
• A configurable limit of simultaneously open files was introduced. This avoids errors caused by the open file limit imposed by the operating system being reached, which can happen in big packs if lots of threads are used.

Codebase changes

• The entire codebase was refactored to be much more modular, testable, and maintainable. In particular, the application and logic code were separated in two binary and library crates, respectively, promoting separation of concerns and paving the way for a future API.
• Automated unit tests were added for the most critical components of PackSquash. This increases the confidence that those components work okay, and continue to work fine after changes. These tests are run automatically for each build.
• Improved build reproducibility by checking in to the repository the Cargo.lock file.
• Updated build environment software. This included updating Linux build runners from Ubuntu 18.04 to 20.04.
• Improved the GitHub Actions workflow, fixing semantic version calculations, build artifacts caching, and other changes.
• Visual Studio Code configuration files were checked in to the repository. This should help new contributors configure their development environment more easily.
• An .editorconfig file was added to improve code readability in the GitHub website.
• Updated libraries and toolchain components for the latest fixes and improvements.

Miscellaneous

• JSON files can now be prettified instead of minified before storing them in the generated ZIP file in a per-file basis. This may be useful for packs under a permissive license that wants to encourage contributions from its users. Similarly, minification of shader and .properties files can now be disabled.
• Opus audio files are now supported as input files in resource packs. They are transcoded to Ogg so that Minecraft can read them.
• Several tidy-ups and improvements to the text that PackSquash outputs. The error message shown when the output path file was a folder was improved to be much more user-friendly.
• Detailed version, author and license information is now shown on startup. More detailed information can be seen with the -v command line switch.
• The hardcoded list of system files that are ignored when ignore_system_and_hidden_files is set to true was expanded and improved.
• Progress messages are now printed to the standard error stream, following the trend and conventions established by other command line applications.
• Official builds for the Linux AArch64/ARM64 platform are now available. Even though these builds should work okay, support for this platform is limited to a best-effort, because PackSquash has not actually been run on this platform on the device of a repository maintainer.
• Information for sponsoring PackSquash developers was added to the README.md file. A FUNDING.yml file was also added so that GitHub displays a relevant "Sponsor" button in its web interface.

Options files changes

• Settings files have been renamed to options files.
• The following options are new (i.e. they were introduced in v0.3.0 and do not have an exact equivalent in v0.2.1):
  • zip_compression_iterations
  • work_around_minecraft_quirks
  • size_increasing_zip_obfuscation
  • percentage_of_zip_structures_tuned_for_obfuscation_discretion
  • never_store_squash_times
  • spooling_buffers_size
  • open_files_limit
  • zip_spec_conformance_level
  • color_quantization_target
  • minify_json
  • delete_bloat_keys
  • maximum_width_and_height
  • minify_shader
  • minify_properties
• The following options were replaced:
  • strict_zip_spec_compliance, by zip_spec_conformance_level
  • quantize_image, by color_quantization_target
• The following options were renamed, with no change in functionality or value types:
  • resource_pack_directory → pack_directory
  • compress_already_compressed_files → recompress_compressed_files
  • allowed_mods → allow_mods