The SILE Typesetter — Simon’s Improved Layout Engine
Some of the changes that came out in v0.14.6 were … less than optimal. The build system changes were for a good purpose, but made life a bit difficult for some distro packagers. I also managed to introduce a parsing regression. It also came to light that a previous API deprecation didn't come with the usual warning and shim for transitional support.
This release is notable for distribution packagers for two reasons. First, there is now an easier way to skip the font dependency check at build time. Second, this release is the first time using LuaJIT is at 100% parity with regular Lua. If your distro has LuaJIT available there is now little reason not to make the switch.¹ The speed increase is substantial, especially for large documents.²
While I was tinkering with smoothing out the build and fixing regressions a couple other fixes landed. @khaledhosny tackled a long standing issue with the font loader. The opsz axis can now be used as expected so fonts with OpentType FeatureVariations are usable in SILE. @raphCode also spotted and fixed some outdated code in the manual.
¹: As far as SILE core and all its Lua dependencies are concerned there is no downside. Previously some of the debug and tracing features did not give as much useful information when using LuaJIT. Now the only caveat is related to end users and their documents. If people write custom Lua code in their projects and/or use other 3rd party Lua libraries they will need to consider interpreter support.
²: As a rough comparison on my desktop machine, rendering the 115 page SILE manual takes about 48 seconds with Lua 5.4, 22 seconds with LuaJIT, and 17 seconds with OpenResty.
This release has tons of fixes and even a few features, but perhaps the most interesting bit to end users might just be the documentation. Thanks to @hegjon, official SILE packages are in the works for Fedora. The procedure for installing them via COPR is documented and will be updated when they land in default repositories. Also @ctrlcctrlv added some useful notes about installing 3rd-party packages. This is really important because very useful actions like typesetting Markdown are best accomplished with external packages.
Anyone working with the source code for SILE itself should note the build system has been split so only run time dependencies are checked at configure time by default. This means no-change is needed for distro packagers and end users that are just wishing to install and move on. However checking for dependencies needed in order to develop SILE itself (including running tests) now requires different configuration. Using --enable-developer should be added as a go-to configure option for anyone hacking or wishing to contribute the to SILE sources. Also of interest to developers, using nix develop should now be a lot more seamless as far as providing a ready-to-use environment.
@Omikhleia has rebuilt the formatNumber() utility function using ICU. This increases language support from a handful of manually implemented ones to almost complete CLDR coverage. In the process a new utility function collatedSort() was added to sort Lua data with locale aware collation. He also fixed a number of little typsetter bugs and overhauled some more documentation.
First-time contributer @raphCode submitted a fix for calling citation keys in the bibtex package.
Debug output is now both faster and more robust since it can't crass the typesetter. When errors are encountered, location reports for where in the document an error was triggered are more accurate. Bundled in this release are also a batch of other small bug fixes to the typesetter, page builder, and shaper.
We're just squishing bugs today and making the manual a little neater. Working with STDIN streams should be a bit more robust now as the content type detection isn't so picky about whitespace and isn't so prone to false positive detections. Document file restrictions have been relaxed a little too allowing comments and whitespace before the leading document tag. Package developers should have a little bit easier time with in the event their package is loaded twice. Also a number of small typesetting issues in the user manual were cleaned up by @Omikhleia.
Today we have a small rollup of fixes and improvements. @Omikhleia overhauled the counters packages as well as fixed up bugs in several other packages. The default center of rotation has been changed to give a more expected result. Text casing functions are accessible again from the Lua side of things, changing masters during a page output doesn't break page breaking, and using the twoside package doesn't force the use of mirroring. @OlivierNicole also stepped in to fixup some math bugs. Also thanks to @ThatGeoGuy and @snan for pointing out and fixing small issues in our documentation.
\unichar'ed chars with same font only (91a8d40)This release addresses a few pain points discovered using the v0.14 series in production.
For 3rd party package developers, it's now easier to find non-Lua resources you installed along with your package. Among several contributions from @Omikhleia, the output backend now returns useful information about images it embeds directly. The same work also allows specific pages of PDFs to to be emdeded as images, not just the first page. He also helped refactor existing interfaces to make them easier to extend, such as the table of contents package. A long standing issue with Turkish hyphenation has been addressed, eliminating the need for preprocessing or other workarounds. More long standing issues with the book class and chapter opening spreads have been addressed. Some priority and interactions between new CLI flags has been worked out. A few more misc bugs were also squished and relevant tests added.
These few quick touch-ups smooth over last minute bugs and ease the transition process from v0.13.x. Besides squashing a pointless warning messages and some bug fixes, this changes the order deprecated includes are processed to facilitate the transition of old projects to new APIs by making it easier to update them one bit at a time.
Work to refactor SILE's internals into a more regular API has been in progress for some time. This release brings a big chunk of that work to light. Many/most of the changes are under the hood and won't show up in the appearance of documents. They do affect anybody writing classes or packages for SILE, hopefully for the better.
Most notably, 3rd party modules have much more robust support. A module may be a class, a package, a typesetter, a shapper, an inputter, an outputter, etc. This makes it easy for developers to extend SILE without needing to modify the core distribution. Modules can be packaged and installed using LuaRocks. LuaRocks.org may be used to host and distribute SILE addons. Rocks may also be distrusted independently. Either way they are easy to install with luarocks install and such installs will make the modules available for loading into SILE. The old built in package manager is still functional for now but deprecated.
The ergonomics of packages has changed a little. The overloaded \script command has gained some more specialized peers. For example Lua code may be passed in \lua commands. To load a package or other module, use \use[module=package.name]. This syntax also allows passing options to modules.
The inputters have been normalized a bit, and XML, SIL, and Lua are all first class input languages. Passing some content in XML format from a SIL format document is as easy as placing it in an \xml command. The same goes for SIL format content in an XML document.
3rd party developers may add new input formats as well! All that is required is an inputter module that returns an AST. Expect to see new inputters soon. @Omikhleia has already been working on a new Markdown inputter module. Other XML based formats such Docbook have been supported for a while, but the new class and inputter APIs make them easier to add and use.
The CLI has gained several new options. As usual sile --help and man sile both have relevant information. Notably the -I/--include argument has been deprecated in favor of more explicit -u/--use, -p/--preamble, and -P/--postamble. As you might expect from those options this makes it possible to load an inputter, class, or package from the command line without having it set in the document. The preamble/postamble system has been cleaned up so content or code can easily be injected at the start or end of a document. This is a change from the previous 'preamble' system that actually became the master document and then included the original master as effectively a postamble.
Many old shims and deprecated functions have been completely removed. Most will still throw relevant warning messages with hints about how to update.
While I was busy moving cheese, several contributers were chipping in new features and fixes. @rolfmblindgren pitched in Norwegian localization and improved hyphenation. @matteodelabre contributed quite a few new features and fixes to the math package. @Omikhleia did some data mining and came up with localization strings for a whole slew of languages. He also pitched in fixes to a number of packages. He also has pending contribution for many new packages and features. Be sure to checkout the PR scene on GitHub if you are looking for new features.
packages: The primary use was probably internal to SILE, but if by chance you have bibtex databases with formatting commands in SIL markup format rather than just plain text content the markup will cease to function and will need to be converted to XML syntax instead. This enables the use of declarative markup in Fluent localizations.
classes: Each SILE package now inherits some interfaces from a common base package. This model allows packages easier access to SILE internals while at the same time tracking what they do so it is easier to enable/disable them. The package knows which document class instance it is attached to, and the document class knows which packages are loaded at any given time. Legacy style packages will continue to work for the time being but will not be tracked in the same way.
core: The role of document commands has always been tightly scoped to classes. For example the book class has a \footnote command while plain does not—unless you manually load the package and setup the frames. In spite of this obvious functional scope, registering commands has been a global operation that stored them in a global registry. In order to allow SILE to be used more programmatically as a library with potentially more than one document and class being processed at at once, these need to be moved out of the global scope. This will also facilitate things like being able unload packages and revert to previous functionality for anything they over-rood on load. For now the functionality is shimmed, but code using the SILE.registerCommand() function should switch to the method of the same name on the current class, i.e. class:registerCommand().
core: Some internal files and APIs got renamed with more structured name spaces. In particular the inputter, shaper, and outputter libraries all have a common naming scheme now and sensible inheritance chains. No functionality was harmed, but if you are overriding undocumented internal Lua methods you might have to update your name spaces to match.
cli: The -I / --include option was overloaded for more than one purpose and is now deprecated in favor of more specific replacements: -r / --require for loading code into SILE before input processing, -p / --preamble for processing content prior to a document and -P / --postamble for processing content after a document.
packages: The original package manager POC that used Git to clone packages into the SILE installation directory has been deprecated. It will continue to function for a while, but all new 3rd party packages should use the LuaRocks based installation process. Whether or not they use luarocks as a package manager or LuaRocks.org as a distribution channel they should install themselves to any usable the system or user Lua library path under a top level "sile" namespace.
classes: The shims allowing classes designed for SILE releases v0.12.x and prior have now been removed and documents using them will now throw errors when rendering. Only the refactored class system introduced in SILE v0.13.0 is supported going forward.
deps: We previously deprecated all use of stdlib. This release stops providing it entirely. If you use it in your own projects you will now need to provide and require() it directly.
build: The C modules compiled as shared libraries (.so files on Linux, .dll on Windows) are now installed to the project root shared directory instead of it's 'core' subdirectory. Distro packages that split the library into its own package will need to adjust this path. People installing from their distro packages or from source should be unaffected, but this will bring us one step closer to being able to install and use SILE as a library.
-r option (a212e83)luarocks (232e72b)\raise / \lower as measurement (#1506) (7196fda)The upcoming v0.14.0 is chock full of changes (hopefully all for the better!) but several small issues have cropped up in the v0.13.x series and there doesn't seem to be any point in inconveniencing existing users with bugs we've already fixed while we work on new features. This should fix up some issues with includes not activating exported functions and getting blocked when both directories and files of the same name exist. The Nix Flake will now work with the new Nix 2.10.x releases.
Additionally @Omikhleia contributed some fixups and features to the Bibtex package to make it much more usable.
This patch release brings both bug fixes and new goodies.
@Omikhleia contributed a new package for formatting lists. Features include bullet, numbered, and mixed list types with many styling options.
@ctrlcctrlv fixed zenkaku unit handling and helped refactor the jplain/jbook classes so we now have tate-capable but language-agnostic tplain/tbook classes in addition to the Japanese specific variants. He also contributed full support for Esperanto including hyphenation, number formatting, and localized output strings. Even if you don't have a use case for Esperanto directly the issues that rooted out with language and class handling should benefit everyone.
The papersize option provided by the base class is now case insensitive and allows dashes. If you prefer A4 to a4 or ANSI-A to ansia feel free to use them. A couple new paper size presets were added in the process.
Additionally the font fallback system got an overhaul. A change from previous behavior is that TOFU handed out by the last available fallback font will be rendered instead of skipping the characters completely. Switching back and forth between the color fonts package and the font fallback package is now possible. Finally it should stop throwing bogus warnings when it successfully shapes a segment via the last available fallback.
Other minor fixes include interactions with color plus dropcaps, some bibtex errors, and errors in the pecha class.
tate and jplain (db83e9e)