Snapraid Aio Script Versions Save

This is a quick fix release to address two issues that have been recently reported after v3.3 release

• TOUCH command not running when using SnapRAID 12.3 #92
• if curl is not installed the script does not correctly check for it, causing to always show there's a new script version #93

Please note: this is a re-release of v3.3.1 I did not update the config file check. Now it's correct, apologies for the confusion.

What's Changed

• Align dev with master by @auanasgheps in https://github.com/auanasgheps/snapraid-aio-script/pull/95
• Push 3.3.1 to master by @auanasgheps in https://github.com/auanasgheps/snapraid-aio-script/pull/96
• Update script-config.sh to match current version by @DonkeeeyKong in https://github.com/auanasgheps/snapraid-aio-script/pull/97

New Contributors

• @DonkeeeyKong made their first contribution in https://github.com/auanasgheps/snapraid-aio-script/pull/97

Full Changelog: https://github.com/auanasgheps/snapraid-aio-script/compare/v.3.3...v3.3.1

A long time has passed since the last release, so there's a lot to like!

• Support multiple script configuration files
  • When running the script you can specify the config file you'd like to use
  • If not specified, the script will use the default config file
• Check if SnapRAID config file exists
  • Alerts the user via email and notifications if it doesn't before exiting
• OpenMediaVault 7 Support #80, #85
  • OMV7 introduced changes to SnapRAID plugin: support for multiple array, new file location and naming
  • The script will search of one file in the new location and use it
  • If there are multiple files, will stop and alert the user
• Update Check
  • The script can let you know when there's an update #67
• Dependencies install
  • install all required dependencies when running a distro that uses apt
• Improvements to Docker management, thanks to @tehniemer #64
  • Local and remote containers can be managed at the same time
  • Can filter containers using wildcards
• Send SnapRAID Status and SMART outputs via notifications on each run - #50
• Improve Discord notification output and its reliability - fixes #55
  • Fix Discord notification not sent in specific conditions - thanks to @Sirver51
• Scrub and Sync errors now are reported as "Severe Warnings" #49
  • In these cases, the notification message tells the user to check logs.
• Detect if SnapRAID is already running before doing anything #62
• Fixed notifications not sent if email fields are empty #56
• Replace awk with bc, fixes errors at the end of emails #74
• Remove unused code for legacy spindown methods
• Other misc improvements, thanks to @tehniemer #64

What's Changed

• Update snapraid-aio-script.sh by @tehniemer in https://github.com/auanasgheps/snapraid-aio-script/pull/48
• Rebase dev docs by @auanasgheps in https://github.com/auanasgheps/snapraid-aio-script/pull/66
• Updates to make docker management more efficient and versitile. by @tehniemer in https://github.com/auanasgheps/snapraid-aio-script/pull/64
• Update script-config.sh by @tehniemer in https://github.com/auanasgheps/snapraid-aio-script/pull/79
• Update snapraid-aio-script.sh by @tehniemer in https://github.com/auanasgheps/snapraid-aio-script/pull/81
• Merge v3.3 from dev in master by @auanasgheps in https://github.com/auanasgheps/snapraid-aio-script/pull/88

Full Changelog: https://github.com/auanasgheps/snapraid-aio-script/compare/v3.2...v.3.3

Here we go with a new round of features and improvements. Thanks to all contributions!

• Script output (log) retention feature
  • You can now keep detailed SnapRAID script output (called log) for a number of days, instead of overriding them at every run.
  • Logs will be rotated and can be stored in any folder you like
• Force zero size sync
  • Forces the operation of syncing a file with zero size that before was not. Useful when handling some system files that genuinely change their size to zero. Use with caution, disabled by default
• New Sync option "added to deleted ratio", thanks to @tehniemer #45
  • Authorize a Sync even if the delete threshold has been breached, but only if the ratio of added to deleted files is greater than the value set.
• Scrub new blocks, thanks to @tehniemer #45
  • New option to scrub data that has just been added to the array
• Customizable Healthchecks.io server URL, thanks to @phidauex #32
• Detailed hd-idle documentation, thanks to @phidauex #35

Upgrade Notice: If you are using a previous version of the script, you will have to move your preferences to the new script-config.sh. The old file can't be reused or the script will throw an error.

This version has been in the works for quite a while. There are many changes, also thanks to the many contributions! The config file has been completely redesigned and is a breaking change for existing users.

• Telegram notifications
  • Send short telegram messages about the script status. It's the same info sent to Healthchecks, which can be enabled at the same time.
  • You will have to create a bot to use this feature.
  • The whole email message does not fit due to space limitations.
• Discord notifications, thanks to @ranapushpender
  • Same feature set as Telegram
• Notification Hook, thanks to @Caedis
  • Allows to use an external service/script/mail binary not natively supported by this script
• Custom Hooks, thanks to @nzlov #27
  • Configure shell commands or scripts to run before and after SnapRAID operations
• Improved error handling
  • Exit with an error when SnapRAID files are not found during sanity check (parity, data)
  • New error messages for email subject and Healthchecks/Telegram
  • If the configuration file is not found, write a message to the default snapraid log location
• Configuration file
  • Configurations are now split in categories
  • You must update the configuration file, moving your configs to the new one
  • If you haven't updated the config file after an upgrade, the script will exit with an error.
  • You must update the configuration time every time is needed. This change ensures consistency and avoids unwanted errors.
• Remote Docker management improvements
  • Add configurable delay for actions to remote host to improve success rates, thanks to @tehniemer #22
  • Manage multiple remote hosts, thanks to @tehniemer #24
• Disks spin down has been fixed, now uses hd-idle
• Migrate python-markdown to python3-markdown, to enable compatibility with Debian 11 / OMV6.
• Fixed email formatting when Scrub job is not run

Upgrade Notice: If you are using a previous version of the script, you will have to move your preferences to the new script-config.sh. The old file can't be reused or the script will throw an error.

The script is back, better than ever! Skipping v2.9 since there have been a lot of changes.

• Email rendering is accurate
  • Tabs, spaces and tables are finally displayed correctly. These sections have unformatted text (code blocs) and will appear slightly different.
• Docker management is back
  • You can choose to pause/unpause or stop/restart your containers
  • If Docker is running on another host, it can be managed too
• Added Healthchecks.io integration
  • Script result can be reported to Healthchecks.io. If the script ends with a WARNING message, it will report a DOWN. Useful to quickly acknowledge failures, since this service can alert you via many channels.
  • Email subject and file summary will be reported to Healtchecks.io
  • Requires curl, which will be installed if not found
• Added "Delayed Scrub" feature
  • Allows to run a scrub job every X runs instead of every time.
• Fixed detection to include all content files
  • Previously only the first file was checked
• Show the number of unchanged files to the summary
• Added more messages to syslog
• Improved many messages
• Code quality: thanks to generous contributions from @cmcginty, a lot has been rewritten and reorganized. The script is more robust, comments have been improved, proper formatting & best practices via shellcheck and more.

Upgrade Notice: If you are using a previous version of the script, you will have to move your preferences to the new script-config.sh. The old file can't be reused or the script will throw errors.

No new features in this release, it's focused on message improvements and general clean-up.

• Added a title for the SnapRAID Status section.
• Changed strings when the output is 0. This nonsense is gone: - The number of deleted/updated files, (0), is below the threshold - The number of threshold warning(s) (0) has reached/exceeded threshold (0) - 0 threshold warning(s) until the next forced sync
• Added a string when using Sync with threshold warning - before was only shown "Forced Sync"
• Moved "Sync is authorized" message for more clarity
• Minor tweaks to other messages
• Removed clean_desc and service_array_setup functions, used for container management

Stuff has been moved around.

• Divided script and config variables two different files
  • Making changes to your configuration is now easier since you don't have to deal anymore with the whole script
• Moved script introduction and changelog to GitHub
• The script will install Markdown if the package is not found
• Moved sync warn file in the same directory of the script. Previously was /tmp which would get cleared after a restart
• Improved threshold messages for clarity
• Added 'snapraid status'. Disabled by default, because the output is a little broken.

First public release here!

Here is the full changelog for all the earlier work that brought to this release.

• version 2.0 changelog by auanasgheps
  • Removed DIFF output from email (still present in logs) - credits to metagliatore
  • Fixed script for Debian 10/OMV5 - credits to sburke
  • Added alternative way of spinning down disks with hdparm
• version 2.1 changelog by auanasgheps
  • disabled disk spindown (code is commented) since it's not working properly
• version 2.2 changelog by auanasgheps
  • Redone changelog section
  • Added 'Prehash Data' feature - credits to Zack Reed
  • Added HTML formatting - NOTE: requires python-markdown
• version 2.3 changelog by auanasgheps
  • Removed TOUCH output from email (still present in logs)
  • Fixed small typos
• version 2.4 changelog by auanasgheps
  • Send an email alert if parity or content files are not found, then exit gracefully
• version 2.5 changelog by auanasgheps
  • Added configurable options for disk spindown and email verbosity
  • Added syslog features
• version 2.6 changelog by auanasgheps
  • Fixed sed error caused by a slash when updated/deleted threshold is breached
  • Fixed violation message not shown when threshold is reached but not exceeded
  • Fixed concurrent (deleted AND updated) violation message and mail subject
  • Added message and mail subject when sync is forced with breached thresholds (deleted, changed or both)
• version 2.6.1 changelog by auanasgheps
  • Disabled clean_desc function in main script, caused {out} file in /root
• version 2.6.2 changelog by auanasgheps
  • Added SnapRAID version to output
  • Added variable for script version
  • Removed timestamps from logs text since it's already added by the system
• version 2.6.3 changelog by auanasgheps
  • Small change to email subject when forcing syncs with violations
• version 2.6.4 changelog by auanasgheps
  • Removed unnecessary capitalized letters in email subject
• version 2.6.5 changelog by ozboss
  • Replaced tabs with spaces
  • Change default of 'SYNC_WARN_THRESHOLD' to '-1', will not force a sync
  • Added requirements
  • Added alternative spindown method (hd-idle)
  • Added automatic detection of rotational devices for spindown (hdparm and hd-idle)
• version 2.6.6 changelog by auanasgheps
  • removed code for Docker Container management, was unreliable