The SILE Typesetter — Simon’s Improved Layout Engine
Hot on the heals of a major release we have small one to patch up some odds and ends. Thanks to @ctrlcctrlv for correcting my oversight in not distributing some of the new localization files. If by chance you've on a new fangled hardware platform like RISC-V things should build out of the box now. The documentation for installation on various systems has also been cleaned up. Thanks to @Omikhleia for some new rule functions and fixes to old ones plus leader alignment corrections.
In the last patch release I hinted we had bigger things coming. This release finally surfaces major internal changes I've been working on since 2019. If our releases had code names this one might be "class wars". The SILE code base has two distinct usages for 'class'. One usage is our document classes, cohesive bundles of features and styles used to render a document. The second usage is the object-oriented programming paradigm of class inheritance. In serendipitous naming ripe for conflation, the document class feature is itself written using programming classes, enabling document classes to inherit from each-other.
This release refactors the implementation of both types of classes. Since nearly its inception, SILE has depended on the stdlib project for many of its Lua abstractions. Unfortunately this library is not as standard as its name suggests, has not been developed or even well maintained, and has been a constant source of problems. It even tinkers with core Lua functionality making it very difficult to keep SILE working across interpreter versions and to debug problems. For some time we've been gradually replacing it with abstractions from the Penlight library (loosely based on Python's standard library). While not perfect, that library of functions is much more robust and predictable. This release finally removes the last of SILE's usage of stdlib. For this release cycle the core program will still provide the stdlib library to ease transition for downstream projects that currently assume it is present, but it is no longer used internally. Beginning with the next major release cycle, v0.14.0, it will not be provided at all. (You will of course be free to include it as a direct Lua dependency in your own projects.)
We've attempted to shim most of the API changes so that most old SILE documents (and in particular, custom classes) continue to work out of the box. Most simple projects should still render without intervention, but you may see many warnings about deprecated functions. A few more advanced projects may run into trouble and fail to render at all, especially if they messed around with classes (of either sort) very much. As far as possible we've tried to add warning notices with hints about how to use the new class models correctly. Please don't hesitate to open issues if you are having trouble getting anything to work. For package authors, a new hook system should make it a lot easier to write packages that do more without also having to write a dedicated class to use them.
Additionally the localization system for all language specific strings that may be rendered by SILE has been changed from the home-grown system of nested SILE commands to a more flexible i10n system based on Fluent. The Lua implementation is not at 100% feature parity with Project Fluent reference implementation, but it is 100% interopperable with other implementations for features implemented and is a much more robust localization system than just substituting strings. Besides using it to easily customize the limited set of embedded localizations (such as the Table of Contents header or chapter titles), the tooling for localizing messages and rendering them in context aware functions is exposed for package developers and document authors to use at will.
A small collection of bug fixes to the typesetter rounds out this release. Justification of lines with ligatured characters is much improved. Rules with a depth property no longer throw off baseline calculations. Empty documents now generate blank PDFs. Adding a new master frameset layout no longer destroys the current page's frameset.
settings: All the functions under SILE.settings.*() should now be called using the instance notation SILE.settings:*(). Usage should be shimmed with a warning for now.
Changing this in your code is relatively easy with a search and replace. As an example with a project in Git, you could use perl like this:
uncs="pushState|popState|declare|reset|toplevelState|get|set|temporarily|wrap"
it ls-files | xargs -n1 perl -i -pne "s#(SILE\.settings)\.($funcs)#\1:\2#g"
typesetter: Making a new instance of the typesetter should now be done by calling SILE.defaultTypesetter() instead of copying the object. It has been changed from a std.object to a Penlight class. As such the correct initialization function is also now _init() instead of init(). A shim is in place to catch legacy usage, but this will be removed in the future.
deps: All calls to the Lua default string library have been using a version monkey-patched by stdlib. This has created all sorts of issues including not being able to properly use some of Lua's default features and conflicts with out explicit meta methods. Also we're busy dropping dependency stdlib altogether.
If you were relying on it for any of your string operations, replace string.func() with std.string.func(). For now std is being provided by SILE, but if you use it in your projects please add it as a direct dependency yourself since that will eventually be removed as well.
By the way in case anything ever git bisects back to here, one way to test if your problem is related to this change or not (especially if you have downstream code that might have built on the assumption SILE's Lua strings were monkey patched) is to load it manually yourself:
sile -e 'require("std.string").monkey_patch()' your_file.sil
classes: This changes the way classes are represented as Lua objects and the mechanism used for inheritance. While shims will be in place to catch most cases that use old syntax it is not possible to grantee 100% API compatibility. If you have classes that do anything remotely fancy (i.e. not just copy/paste from SILE examples) they may or may not work at all; and even if they do they should be updated to explicitly use the new API.
Today we have a fairly small release motivated primarily by wanting to patch a few small issues before some bigger things in the pipeline land. We cleaned up a few small scripting issues with the configure and install process that should make it easier to install from sources on BSD platforms or Linux distros such as Alpine that do not default to bash and GNU tooling.
Besides the minor tooling fixes, @Omikhleia has continued to pour in contributions. This release features his feature overhaul to the leaders package so that dot fills align vertically. It also includes his fixes to whitespace handling around punctuation in French and better extraction of strings such as chapter titles to PDF bookmarks.
make selfcheck (319e0c5)This patch release fixes a bug introduced while fixing other bugs. While enabling BiDi by default for all typesetters was working, enabling it manually on typesetters would result in an infinite loop. Given that many people worked around it not being enabled by default by turning it on manually, this meant a lot of documents started taking lots (∞) of time to render. Apologies for the wasted cycles, whether CPU or brain wave.
This patch release changes the way new frames are generated to inherit settings differently. These changes should qualify as "fixes", but could affect workarounds for anybody that had trouble before. Loading the BiDi package (which happens by default) now changes the default typesetter in addition to the current one. This affects some features such as the commands to typeset into a different frame than the current one when RTL languages are involved. Additionally, the way frames are created has changed so that direction settings such RTL set on the class are inherited by default to default and new frames unless otherwise specified. This affects folio, footnote, and other default frames as well as any created programmatically.
Both changes are likely the behavior most users would expect anyway –hence the minor patch release– but could cause unexpected results in documents that had other ways of dealing with the previous unexpected behavior.
This patch release is to fix a regression in handling negative width spaces — something that turn up when typesetting Nastaliq. Thanks to first-time contributor @mimbrown for both the report and fix. Also the documentation for the dropcap package added in the last release is now in the manual. The only other thing possibly worth noting is that the prebuilt Docker images now run on the recently released Lua 5.4.4.
The first minor release in the 12.x series brings several new features relevant to paragraph shaping. @Omikhleia has fleshed out the linebreak mechanism with full parity of LaTeX's settings including hangingIndent and parShape. Having fully featured paragraph-wise hanging indent settings enabled a new package, dropcaps, to provide much more robust dropcap functionality than the previous floating frame mechanism we used for this. (Note the dropcap package documentation did not make it in the manual, see the example on the website for usage.) Additionally the paragraph shape callback function can now be used to create fancy paragraph shapes using any geometry you can specify or calculate. Bugfixes in this release cover several issues with French language, PDF metadata encoding, color fonts, book class headers, and more.
While this release follows close on the heels of previous releases it represents over 2 years of work from community contributors. @harrysummer started work on a native math feature in April of 2019. While this initial phase was never completed, it did get as far as a working prototype. In November of 2020, @OlivierNicole picked up on the work and transformed the prototype into the shape as found in this release. Additional thanks to @akavel for input throughout the process.
On a conceptual level the math support is based on MathML and relies on OpenType math fonts. The input notation may be MathML (either directly using the native XML syntax or using an 1-to-1 equivalent SILE syntax) or an alternative math shorthand similar to the math notation used by TeX. See section 5.29.5 in the v0.12.0 manual for detailed documentation on the new math package. Examples are given for various formulas and syntax options. A math example has also been added to the packages example gallery on this website.
Additionally @Omikhleia contributed several new features and fixes including much more control over Table of Contents, styling options for PDF links, correct French punctuation treatment, and more.
Thanks to the contributions of @doronbehar, as of this release SILE is usable as a Nix Flake (in addition to the usual Nix package). This is an easy way to test run the latest Git HEAD version at any time from almost any OS. One you have flake support enabled, just run nix run github:sile-typesetter/sile -- <sile orguments>. This website uses this to regenerate some of the examples in the gallery.
Even though the math package is a major new feature, ultimately the major version bump was caused by a breaking change elsewhere. Previous to this release, the font used for footnotes was the SILE default unless specifically set. Setting changes were relative to the SILE default rather than the document default. This behavior was workable, but usually came as a surprise to most users. With this release, footnote and folio frame font settings default to being relative to the current document font. The font family, size, leading, and all other aspects of footnotes and folios are still configurable with the same tools as before. But this is a breaking change because those settings are now relative to a different base which might cause many documents to need updating.
- with − in numbers (f8d490c)classes: Don't increment counters on unnumbered book sections (6cfca4d)
core: Correct --help output to reflect required values (da487ec)
languages: Add test 704 for French punctuations, fix expected 621 and 702 results (8e9b056)
languages: Correct Armenian support to use ISO 639 code ‘hy’ (ffafbe6)
languages: Correct punctuation rules for French (95c2398)
languages: Don't initialize Japanese unless actually called for (3aba931)
languages: Shortcut ICU soft breaks in French (ed8734c)
math: Fix underover error with sub wider than base but no sup (bc87393)
packages: Don't replace shaper unless actually initializing color-fonts package (269ca59)
packages: Fix deprecation warning command in package docs (a69d774)
packages: Reset footnote and folio settings top level state (3795a4e)
shaper: Fix memory leak in Harfbuzz library (#1243) (035dcc8)
Today's small patch release is mostly to help out downstream packagers with some small issues that cropped up. First it's a stab in the dark hoping we made things easier to get Homebrew packaging working on ARM based macOS. Secondly we added a build time check for git, which is required in order to use SILE's internal package manager. This dependency may change in the future, but since we use it now it's helpful that system compatibility is checked at build time. Lastly we fixed up some internal usage of an API we deprecated 6 releases ago and documented as going away in v0.11.0, but forgot to stop using ourselves. Notably the French language support module was affected.
This release brings a significant fix to the vertical grid typesetting system that was causing extra full grid spaces between paragraphs where they didn't belong. The new behavior is objectively more correct, but it does break previous workarounds. If you previous used a workaround to get the right output in spite of the bug, those documents may not render incorrectly. They can be fixed by removing whatever other workaround was in place or adding a shim that makes the paragraph spacing emulate the old behavior. Some utility functions also received bug fixes. Thanks to contributor @ctrlcctrlv, color fonts now behave themselves much better in TTB layouts and the counter display style Alpha works as documented.
Not much has been added on the feature front, just a few conveniences such as traditional units of (small) measurement and handling more attributes in documents converted via the Pandoc writer (as-yet unreleased).
Beginning with this release Docker images will now be available from GitHub Packages (also known as the GitHub Container Registry). This is an alternative to Docker Hub which has recently started limiting the features available to open source projects.
packages: Previous to this release any and all leading between paragraphs (as set with document.parskip) –even a 0 height skip– would result in the skip of one full empty grid space — as if parskip had been set to something approximating a full line height. This change corrects the calculation so if a 0 height skip is added and everything fits, the next line or paragraph will continue uninterrupted in the next grid slot. To get the previous layout behavior back, document.parskip must be explicitly set to be something larger than 0. Even a minimal 1pt skip will result in paragraph spacing that includes one full grid height left blank as before:
\set[parameter=document.parskip,value=1lh]
utilities: Previous return value for breadcrumbs:contains() was just an depth index with -1 indicating no match. This made sense when I wrote it, but coming back to it for a new project I expected a boolean return value. Returning two values seems like the best option, but given the function naming it seemed to make sense to return the boolean first, hence the API breakage.
utilities: Change breadcrumbs:contains() to return <bool, index> (a987394)