Modernish Versions Save

The installer can now bundle a reduced copy of modernish with your scripts, so they can run portably with a known version of modernish without requiring prior installation.

This version should not break existing modernish scripts. The new features selected for backporting to this release seemed like necessary improvements because they fix important shortcomings in modernish. From now on, development of new features happens on the master branch and 0.16.x only gets bugfixes. I mean it this time :)

This version is tested on AIX (7.1), Cygwin, DragonFlyBSD, FreeBSD, HP-UX (B.11.11), Illumos (DilOS, OmniOS, OpenIndiana), Interix, Linux (Debian, Gentoo, NixOS, Slackware, Ubuntu), macOS / Mac OS X, NetBSD, OpenBSD, QNX (6.5), and Solaris (10.1-11.4). Testing on other systems is wanted!

Some of the less common UNIX systems' default shells are not compliant with the latest POSIX spec. If modernish reports fatal shell bugs, make an alternative shell available in $PATH.

Differences between 0.16.3 and 0.16.5

Below is a summary of changes, optimisations and fixes. See the git log for the complete overview with explanations: git clone -b 0.16 https://github.com/modernish/modernish cd modernish git log -p --reverse v0.16.3..v0.16.5

Changed:

• NEW: install.sh -B: bundle the modernish library with your scripts. This version adds a -B (Bundle) option to install.sh which allows bundling any number of scripts with an reduced copy of modernish into a separate directory tree (indicated using the -D option) that is ready to run, archive and distribute. This allows scripts to run immediately and portably with a known version of modernish, even on systems that don't have modernish installed. You can indicate a preferred shell with the -s option or you can have your script be shell-agnostic. Even if a preferred shell is indicated, modernish will fall back to other shells if the preferred one isn't available. It is left to the script to check for required features (typically with thishellhas). Details: README.md, Appendix F

• var/loop/find: Enhancements for compatibility and the use of nonstandard primaries:

  • Now allows specifying, as an argument to use var/loop/find, a preferred find utility to use for the find loop. For example: use var/loop/find bsdfind # prefer utility by this name use var/loop/find /opt/local/bin # look for a utility here first use var/loop/find /opt/bin/gfind # try this one first
  • New --try option: try an expression primary. It makes "unknown primary" errors non-fatal: upon encountering a primary that is not supported by the utility, the loop will quietly abort with exit status 128, leaving the -name of the invalid primary in the REPLY variable. This allows a script to test for the presence of certain primaries and decide what to do.
  • New optional compatibility mode for obsolete/broken find utilities. Passing the -b or -B option to use var/loop/find allows broken find as a last resort; modernish will fall back to older and very inefficient methods using -exec ... {} \;.

• var/local, var/loop/for, var/loop/select, var/loop/find: New --base option: prefix a string/directory before doing --split or --glob. This provides a way to have an untrusted prefix part of a path or string not subject to split or glob while doing these expansions in safe mode. For example, this is more secure when expanding wildcards on files within a base directory path input by the user, or passed from untrusted data. Rationale: 739c959

• The var/mapr module is renamed to sys/cmd/mapr. A compatibility redirect is left so that use var/mapr will quietly continue to work in modernish 0.16.x. After 0.16.x, this will display a warning; still later, it will be removed. Rationale: 92fea83

  • In addition, sys/cmd/mapr now depends on sys/cmd/procsubst, and no longer depends on sys/cmd/extern.

• sys/cmd/harden: The harden function now will not harden read. Rationale: 62862b2

• thisshellhas:

  • There are no more "internal" bug/feature/quirk detection tests; all are now implemented as cap/*.t detection scripts.
  • The output of thisshellhas --show is now sorted.

• Initialisation for non-interactive shells (i.e. scripts):

  • Ensure IFS (field splitting) has a standard default value; this is to counteract /bin/sh (dash) on Debian-based systems inheriting IFS from the environment, which is a glaring attack vector on those systems.
  • Avoid inheriting CDPATH, so that the cd builtin is not influenced.

• New shell features detected: PROCREDIR, PROCSUBST, TESTERE

• New shell bugs detected: BUG_ALIASCSHD, BUG_ALIASPOSX, BUG_CDPCANON, BUG_CSUBRMLF, BUG_DEVTTY, BUG_LNNONEG, BUG_PSUBIFSNW, BUG_PSUBIFSWH, BUG_SHIFTERR0, BUG_ZSHNAMES, BUG_ZSHNAMES2

Optimised:

• var/loop: use PROCSUBST (bash/ksh/zsh process substitution) or PROCREDIR (yash process redirection) where available. This makes loop entry faster on shells with one of these features, and more reliable on buggy kernels with a FIFO-related race condition. Details: 4279d67 92844ef

• var/loop/find: LOOP find handles loop entry considerably faster and generates loop iterations more efficiently. Details: 95417d4

• shellquote:

  • Improved size optimisation for multiple levels of quoting.
  • Allow full size optimisation for shellquote -P. Details: 98e0848

Fixed:

• Modernish is now tested on two new systems:

  • AIX 7.1.0. Requires GNU utilities (from "AIX Toolbox for Linux Applications") in /opt/freeware/bin; its default utilities are not up to the recent POSIX standard. On AIX, modernish now prefixes /opt/freeware/bin to $DEFPATH.
  • HP-UX B.11.11. Requires adding a recent POSIX-compliant shell. Thanks to polarhome.com for providing shell accounts on these operating systems.

• Modernish 0.16.x will now once again initialise on ancient mksh versions with BUG_NOOCTAL + BUG_CMDSPEXIT (such as mksh R44).

• use: When querying (-q or -e options), use now doesn`t throw a fatal error on encountering an invalid module name, but returns exit status 2 instead.

• thisshellhas:

  • Fixed: On bash versions compiled with "minimal configuration", thisshellhas falsely detected a builtin command if a readonly function by that name was set.
  • Fixed: On mksh <= R50f (with BUG_CMDVRESV), thisshellhas had become incapable of detecting shell reserved words/keywords.

• shellquote: Security fix: in "double" quoting, control characters following a $, `, , or " were not translated to $CC* expansions, so a quoted string could still span lines.

• str ematch: awk fallback version: add bug-compatibility with Busybox awk. Details: d188787

• On bash 5.0 and later, modernish now sets the localvar_unset shopt on initialisation, so the shell no longer has the treacherous QRK_LOCALUNS2 shell quirk.

• Bash on NetBSD crashes when detecting DBLBRACKETERE on NetBSD in a UTF-8 locale. Fix: make this crash non-fatal for modernish. Details: 4ff76e8

• The PSREPLACE feature is broken on mksh versions <= R51. At least $CCa (\7) fails to be replaced. Modernish now will not detect this feature on these mksh versions. Among other things, this fixes shellquote on them.

• sys/base/readlink:

  • Fixed: on macOS, readlink died with Internal Error 2 on a symlink with no read permission (which macOS enforces).
  • Fixed: readlink -m was canonicalising incorrectly when traversing through a recursive symlink as a pathname component.
  • Fixed: readlink -f/-e/-m did not resolve symlinks in the $PWD when given a current-dir filename like foo or ./foo.

• sys/base/tac:

  • Fix -r: accept a proper POSIX Extended Regular Expression as documented (instead of an awk regular expression). Details: 57e661d
  • Bugfix: a non-regex field separator of \ was broken.
  • Bugfix: a single-character field separator was not interpreted as a regular expression even if -r was given.
  • Bugfix: tac failed to throw a fatal error if awk failed.

• sys/base/which: which -a now avoids duplicate output for duplicate $PATH entries.

• sys/cmd/extern, sys/cmd/harden: Fixed: Empty $PATH elements were not correctly handled as equivalent to . (the present working directory).

• sys/cmd/mapr (formerly var/mapr): Fixed: Severe bug that caused data loss in an input steam > 4 MB. Details: dae120b

• var/loop/find: Fixed use of -ok and -okdir in LOOP find:

  • Fixed: If -iterate was used twice after -ok or -okdir within the same group of parentheses, LOOP find would die with a syntax error.
  • Fixed: When -ok or -okdir are used, the loop disables an internal optimisation so it iterates immediately after positive user input. This effect is no longer limited to the current set of parentheses.
  • LOOP find now correctly handles the -ok somecommand {} + extension that some find utilities support.

• var/loop/find: Init now rejects find utilities that cannot correctly handle more than one -exec ... {} + clause (which caused multiple -iterate primaries to malfunction). Utilities with this bug include the standard find on HP-UX B.11.11.

• var/stack/trap, var/local: Fixes for use on an interactive shell. Details: a111a3c

• Fixes for install.sh:

  • Now require the user to confirm the response with the Return key for interactive yes/no questions. Before, install.sh used sys/term/readkey to only require a single keystroke. But that was doing more harm than good: users might still press Return and accidentally give the default answer to the next question.
  • Fixed: Relaunching the installer to run with another shell failed if the present working directory was outside of the source directory.
  • Install all top-level documentation files into share/doc/modernish, including EXAMPLES.md.
  • For bash, add -p to hashbang path in bin/modernish so portable-form scripts aren't at risk from inheriting exported functions from the environment.
  • Various fixes for safe directory handling for the -D (destdir) option.
  • Fix handling of relative path arguments on the command line.
  • Allow installing in directories pointed to by symlinks.
  • Cygwin compatibility fix: Avoid double initial slash if the installation root directory or $HOME is /.
  • Fixed: due to a typo, the $MSH_SHELL path was not ensured to be absolute.
  • Can now be invoked by /bin/sh on Solaris 10 and other ancient Bourne shells, as long as a POSIX shell is in getconf PATH.