24.04.1

We are proud to announce version 24.04.1 of Subiquity, supported on Ubuntu 24.04 LTS, Ubuntu 23.10, Ubuntu 22.04.4 LTS, and Ubuntu 20.04.6 LTS. Subiquity 24.04.1 is included in the Ubuntu 24.04 LTS live-server ISO, and is now live on the stable channel as a snap refresh for the other supported ISOs. Also, Ubuntu Desktop Bootstrap on Ubuntu 24.04 LTS includes this same version of Subiquity.

What's new in Subiquity 24.04.1

Autoinstall support for Desktop and Seven Ubuntu Flavors

Ubuntu 24.04 LTS marks the first LTS release of the Subiquity-based installer Ubuntu Desktop Bootstrap, for not just Ubuntu Desktop, but also many Ubuntu flavors - Edubuntu, Ubuntu Budgie, Ubuntu Cinnamon, Ubuntu Kylin, Ubuntu Mate, Ubuntu Studio, and Xubuntu. This also means that autoinstall is now available for all these environments with Long-Term Support.

ZFS With Encryption support

ZFS Guided storage has been extended to support full-disk encryption. This uses the same method of handling the encryption employed by Ubiquity, where a combination of a LUKS keystore + ZFS Native encryption is employed.

While ZFS with or without encryption is not yet offered in the server installer UI, it is available via autoinstall for all Subiquity-based install ISOs including Ubuntu Server:

autoinstall:
    storage:
        layout:
            name: zfs
            password: passw0rd

To change the passphrase, consider using gnome-disks or cryptsetup luksChangeKey.

Additionally for ZFS support, a dataset for /home is now created.

Improved Presentation of Autoinstall Errors

Subiquity's error reporting and Autoinstall validation mechanisms have been significantly reworked to provide clearer and more actionable feedback when encountering errors. Automatic crash report generation has been disabled for cases of preventable errors which are caused by user-provided data (e.g., failure to validate the Autoinstall configuration) and instead helpful error messaging is shown to facilitate resolving the situation.

New Error Overlay

During interactive server installs, a new error overlay has been introduced to provide details on these preventable errors which inhibit the install from proceeding as expected. Some examples below:

• When the provided Autoinstall config fails to validate, the installer halts and informs the user of the offending section:

• When the installer is unable to find a disk matching the provided match directive from the Autoinstall config, the installation is halted:

Improved Event Reporting

Similar to the new error overlay, the event reporter provides details on the error which caused the install to halt. This provides a similar experience for non-interactive installs:

• The provided Autoinstall config fails to validate:

  ... start: subiquity/Keyboard/load_autoinstall_data: finish: subiquity/Keyboard/load_autoinstall_config: Malformed autoinstall in 'keyboard' section finish: subiquity/load_autoinstall_config: Malformed autoinstall in 'keyboard' section An error occurred. Press enter to start a shell

Additionally, the event reporter now includes log-like event types to provide improved visibility on progress of the installation. One such event - the warning event - may be used to report errors which are not critical, but may cause the installation to proceed differently than desired or expected.

For example: misspelling interactive-sections may cause the install to proceed in a fully automated fashion, where a partially interactive install was likely desired:

warning: subiquity/core_validation/top_level_keys: Unrecognized top-level key 'interactive-section'

Smarter Delivery Methods

Subiquity has improved the way it consumes Autoinstall. Now when Autoinstall is delivered via cloud-config, Subiquity will halt the install with an error if it detects autoinstall keywords outside of a top-level autoinstall keyword, instead of silently ignoring them. Subiquity will also halt with an error if the Autoinstall config on the installation media is attempted to be used as a cloud-init datasource.

Consistent Autoinstall Formatting

The Autoinstall configuration file format has been expanded to now support the top-level autoinstall keyword in all delivery methods. Previously, this keyword was only used when providing Autoinstall via cloud-config. It is now advised Autoinstall configurations follow the following format:

autoinstall:
    version: 1
    identity:
        ...

The existing format for autoinstall on the install media (version and other keywords at the top level) remains supported and there are no plans for deprecation. See the new Autoinstall delivery tutorial to read more.

Experimental support for NVMe over TCP installations

Subiquity grew partial, experimental, support for installations targeting remote storage devices using NVMe over TCP. Remote storage devices can be used to store filesystems that are not essential for booting (e.g., /home, /srv). They can be used to store the root filesystem with the following limitations:

• only basic network configurations are supported (ethernet with static addressing or DHCP)
• the /boot filesystem and the ESP must be stored on local storage

There is no screen or specific autoinstall support for discovering NVMe/TCP storage. One can install the nvme-cli package in the live environment and then run nvme connect (or nvme connect-all) commands to perform the discovery. NOTE that it is possible to run these commands as part of early-commands to automate the process.

Reworked SSH screen

The SSH screen was reworked so that multiple SSH identities can be imported (from Launchpad and/or from Github). The keys are displayed in a list and can be selectively previewed or removed. This also prevents keys imported using autoinstall directives to be discarded when ssh is marked as an interactive section.

Changed behavior after failed mirror connectivity check

After a failed mirror connectivity check, automated installations will now revert to semi-offline by default. In a semi-offline installation, packages are only fetched from the installation media and never from the archive. The behavior from previous versions can be obtained using the apt: fallback: abort directive.

For interactive installations, the option to continue despite the connectivity failure is now gone. Instead, the user must decide between doing a semi-offline installation (see above) or going back and adjust network and/or mirror settings.

Status of migration of documentation to Read the Docs

Read the docs is now the preferred location of Subiquity install documentation, and the existing Server install guide copy is being deprecated. The source for this documentation, in reStructuredText format, can be found in the doc directory of Subiquity on Github.

Bugfixes

• LP: #2039966 (PR: 1846) - Update instructions on how to test local Subiquity or related changes in kvm
• (PR: 1857) - Fix dependency list for python3-debian in apt-deps.txt
• (PR: 1880) - Accept a fixed partition size for reset partition in autoinstall
• (PR: 1882) - The sample autoinstall-user-data found at /var/log/installer/autoinstall-user-data now includes a link to the documentation
• (PR: 1885) - Apt: Only backup/restore sources.list if it exists
• (PR: 1888) - Improve text of reported bugs during a failure in curtin by showing which stage failed
• (PR: 1894) - Factory-reset: remove EFI boot entry, fix GRUB entry
• LP: #2051586 (PR: 1914) - Fix crash when attempting to edit a bond with a valid transmit hash policy.
• (PR: 1915) - Fix detection of Netplan on recent Ubuntu releases.
• LP: #2055077 (PR: 1923) - Fix an issue where a call by subiquity to cloud-init could cause reprobing of datasources
• LP: #2037038 (PR: 1931) - Ensure the Distribution Channel Descriptor is copied to target.
• LP: #2051338 (PR: 1934) - Fix issues with match directives and install to RAIDs. Note that there must be a known path to making that RAID bootable for this to work, and not all RAIDs are known to be bootable today.
• LP: #2056783 (PR: 1938) - Fix documentation of match specs for model and vendor
• LP: #2059134 (PR: 1950) - Fix creation of unformatted partitions from ubuntu-desktop-bootstrap
• LP: #2053002 (PR: 1952) - Cleanup zzzz-temp-installer-unattended-upgrade to ensure it is not present on the target system.
• LP: #2059389 (PR: 1957) - Do not suggest resize of a partition that is already mounted.
• LP: #2045280 (PR: 1961) - Fix UI crash when size of partition gets capped.
• LP: #2060695 (PR: 1970) - Workaround an issue with the cloud-init cc_dpkg_grub module by disabling it for subiquity-installed systems. In this case, the system is already bootable, so the cc_dpkg_grub module should be redundant. If cc_dpkg_grub is still required, see this bug comment for suggestions.
• (PR: 1974) - For desktop installs, retain the quiet splash grub GRUB_CMDLINE_LINUX_DEFAULT values
• curtin: LP: #2037682 (MP: 460960) - Fix chroot detection for curtin in-target
• curtin: LP: #2016860 (MP: 462753) - Fix wipefs failure

Known Issues

• Netbooting the Subiquity-based Desktop installer images causes the installer to crash on startup. The issue will be resolved for the 24.08.1 release, or sooner, of Subiquity and at that time will be available via a manual snap refresh in the live environment on older images (LP: #2062988).
• In presence of NVMe storage devices, if the initial block probing operation fails (e.g., because of a timeout), the restricted probe will also fail with: ValueError: Invalid dep_id (nvme-controller-nvme0) not in storage config (LP: #2063162).

New Contributors

• @rkratky made their first contribution in https://github.com/canonical/subiquity/pull/1822
• @icnahom made their first contribution in https://github.com/canonical/subiquity/pull/1857
• @alfonsosanchezbeato made their first contribution in https://github.com/canonical/subiquity/pull/1881
• @julian-klode made their first contribution in https://github.com/canonical/subiquity/pull/1885
• @medicalwei made their first contribution in https://github.com/canonical/subiquity/pull/1880

Full Changelog: https://github.com/canonical/subiquity/compare/24.02.1...24.04.1

• LP:#2040654 - fix a regression in behavior of late-commands using curtin in-target and similar constructs.
• LP:#2040190 - merge a Curtin fix that addresses efibootmgr related install failures that may occur on systems using "EFI v1.1 by Apple" or similar.
• LP:#2052524 - fix warning from Netplan about permissions. In most cases this is just a warning, but if non-wifi secrets were written in the Netplan YAML then they would be visible to non-root users.
• Text corrections related to Ubuntu Pro, SSH, and Identity.

What’s changed in Subiquity 23.10.1

TPM-backed full disk encryption

Particularly notable in this release is the support for TPM-backed full disk encryption. See the Ubuntu blog for details. Note that TPMFDE requires ISOs that are constructed with this feature in mind, which only applies to the mantic-desktop ISO today.

TPMFDE can be triggered with autoinstall:

storage:
  layout:
    name: hybrid
    encrypted: yes

Guided ZFS

New in this release is the support for Guided ZFS, which is available for Desktop and Autoinstall. The guided ZFS structure is brought from Ubiquity and is very similar. (Encrypted Guided ZFS support is in plan for 24.04.)

To use ZFS from autoinstall, configure:

storage:
  layout:
    name: zfs

For Server installs, a recovery key option is now available.

Bugfixes

• CVE-2023-5182 (PR: 1820) - Address CVE by ensuring the autoinstall-user-data file is not readable by the adm group. The file contains the hashed password of a user with sudo capability.
• LP: #2009141 (PR: 1806) - Rework udev events handling in the storage code so that Subiquity does not crash if a large burst of events occurs.
• LP: #2037133 (PR: 1807) - Fix incorrect documentation. The locale is not implicitly made interactive when another section is marked interactive.
• LP: #2032961 (PR: 1811) - Ensure we run early and late commands with a clean environment to avoid potential failures in focal.
• LP: #2017278 (PR: 1812) - Fix crash when attempting to install ubuntu-restricted-addons during an offline install. Subiquity will now warn and skip the package instead. An additional change in the desktop installer was done to ensure that the installation is not wrongly considered offline in some scenarios.
• LP: #2017862 (Probert PR: 136, Curtin MP: 442225) - Fix partitioning failure when existing GPT partitions have UTF-8 partition names
• LP: #2039084 (PR: 1669, Curtin MP: 441769) - Fix partitioning failure when a swap partition is placed in an extended partition
• LP: #2018280 (PR: 1671) - Ensure curtin does not reorder UEFI boot entries since it is unsupported on Dell XPS and some other platforms
• PR: 1687 - Fix crash when a cloud-config is sent that adjusts the users but creates no default user
• LP: #2023265 (PR: 1688) - Fix list of third-party drivers considered empty when it has not yet finished being retrieved
• LP: #2029028 (PR: 1701) - Prevent Subiquity from discarding configuration in /etc/apt being deployed by deb packages
• PR: 1718 - Fix invalid source ID in documentation (ubuntu-server-minimized → ubuntu-server-minimal)
• PR: 1747 - Ensure that running snap start subiquity.probert works as intended
• PR: 1777 - Fix crash when refreshing the installer as part of an automated installation
• LP: #2039076 (PR: 1780) - Fix wrong order of lower layers when building an overlay – resulting in inconsistencies when listing third-party drivers
• LP: #2039075 (PR: 1782) - Fix invalid use of x-error-msg when the error message contains a newline character
• LP: #2039086 (PR: 1786) - Fix crash occurring upon accessing the Help menu after creating a bond interface
• LP: #2034270 (PR: 1787) - Fix crash when using guided storage with a small disk
• LP: #2012659 (PR: 1789) - Fix crash when a Wi-Fi or Ethernet interface gets removed from the system
• PR: 1796 - Fix crash when reusing an existing RAID
• LP: #2039090 (PR: 1797) - Disable unattended-upgrade minimal steps to speed up the installation
• LP: #2035034 (Curtin MP: 451298) - Fix curtin failing to handle partitions on 4k sectors disks
• LP: #2039080 (PR: 1808) - Fix race condition sometimes causing an early crash when using the desktop installer
• LP: #2039082 (PR: 1814) - Prevent Subiquity from restarting costly operations when selecting the same source (e.g., ubuntu-desktop-minimal) multiple times.
• PR: 1819 - Ensure cloud-init logs on the target system are owned by root since syslog:adm is not necessarily the same as that of the installer environment
• LP: #1933055 (PR: 1757) - Ensure the client saves the TUI mode (i.e., rich vs monochrome), so that it is restored after updating the installer

Bugfixes that are already in the stable channel but new since 23.04

• LP: #2033977 (PR: 1790) - Ensure the security archive is set to the proper URL
• LP: #2013201 (PR: 1759) - When setting the size of a partition, Subiquity may crash with an error about "size or offset not aligned"

New Contributors

• @ITJamie made their first contribution in https://github.com/canonical/subiquity/pull/1536
• @s-makin made their first contribution in https://github.com/canonical/subiquity/pull/1685
• @lindhe made their first contribution in https://github.com/canonical/subiquity/pull/1718
• @rkratky made their first contribution in https://github.com/canonical/subiquity/pull/1822

Full Changelog: https://github.com/canonical/subiquity/compare/23.04.2...23.10.1

This release addresses an issue with amd64 installs (LP: #2033977) where the security pocket in sources.list can end up not configured matching documented best practices. It is recommended to configure the $series-updates to come from security.ubuntu.com, to minimize the effect of mirroring delays.

Affected systems will be AMD64 systems installed with Ubuntu 23.04, 22.04.3LTS, or installs done since April 2023 that accepted the snap version update.

To correct systems already installed, please modify /etc/apt/sources.list, look for the lines containing -security, and update them to use http://security.ubuntu.com.

"Ports" architectures (everyone but amd64) should not be affected by the above issue and should pull security updates from http://ports.ubuntu.com

This release was done to address an issue with manual partitioning.

Fixes:

• LP: #2013201 - When setting the size of a partition, Subiquity may crash with an error about "size or offset not aligned"

Known issues:

• 22.04.3 LTS live-server, which contains cloud-init version 22.2.1-0ubuntu0~22.04.1, is affected by bug https://github.com/canonical/cloud-init/issues/4271. The effect of this is that, when using cloud-init to provide autoinstall data, the h aka local-hostname or i aka instance-id nocloud datastore arguments should not be used. For an example of a working configuration, please see the autoinstall-quickstart guide.

Version 23.04.3 is a rebuild to pick up new dependencies in the Subiquity snap. It is source-code identical to 23.04.2.

Fixes:

• LP: #2022102 for the upcoming Ubuntu Server LTS 22.04.3, an issue where the autoinstall data may fail to be seen.

What's changed in Subiquity 23.04.2

The primary theme for this release was productization fixes in support of Ubuntu Desktop Installer. This includes many partitioning enhancements, fixes to enable autoinstall for Ubuntu Desktop Installer, and improvements in handling of NetworkManager.

Another item is improvements in handling of Apt. Many Subiquity failure reports have the Subiquity-Apt interaction as the cause. We now more proactively check that the chosen mirror looks valid, offer autoinstall controls to allow the install to continue if the mirror cannot be contacted, and more. On similar note, a failure to complete unattended-upgrades is no longer fatal to the install.

On the subject of offline installs, there is a new fallback key to control the desired behavior when the Ubuntu archive mirror is unavailable during install time. The default policy is abort, which ends the install if the archive mirror is unaccessible. In some situations, it is acceptable to proceed with an offline install when the mirror is inaccessible. In this scenario, it is advised to use:

apt:
  fallback: offline-install

A common feature request is to adjust how LVM uses a full disk during guided storage creation. It is by design that the rootfs will not use all available VG space by default, to allow creation of further volumes after install, snapshot creation, and generally take advantage of LVM features. However, today LUKS encryption is only offered on LVM, so users desiring full disk encryption would be left with the rootfs not actually using the full disk, until a later expansion of the volume post install.

To this end an autoinstall policy has been added to optionally allow expanding the LVM rootfs LV to use the full disk. Ubuntu-desktop-installer has this expansion on by default, while Server retains the existing scaled usage. When installing via the Subiquity TUI mind that the size of the created volume may be adjusted on the confirmation page. Further improvements around disk encryption, guided installs, and volume handling are anticipated in upcoming releases.

On the subject of LVM, encrypted installs may now be autoinstalled. While we have taken care to not leak the password to the logs in this case, mind that transport of the autoinstall data may reveal the encrypted password. Please consider changing this LUKS passphrase after install is complete if you are taking advantage of autoinstall for a LUKS installation.

The keyboard screen has seen an improvement where the chosen language is used to select a different default keyboard layout.

Several documentation fixes and updates have been done. They are tracked in source code now, allowing for enforcing that future autoinstall changes have a matching autoinstall reference update.

Known issues:

• If using autoinstall to configure network for Desktop images with Ubuntu Desktop Installer, mind that NetworkManager is in use, so configure with renderer: NetworkManager.

The 23.04 Ubuntu Server ISOs have been built with Subiquity 23.04.1. The specific differences between 23.04.1 and 23.04.2 are:

• LP: #2016280 (PR: 1659) - Fix an issue with Subiquity 23.04.1 on Ubuntu 20.04.x where apt fails to run due to library conflicts
• LP: #2002413 (PR: 1652) - Fix a crash issue with Ubuntu Desktop Installer that could occur when editing a swap partition
• (Probert PR: 133) - Fix issue with NTFS partitions where ntfsresize --info was showing progress output in the logs
• LP: #2016901 (PR: 1660) - Fix an issue in Ubuntu Desktop Installer where manual partition edits could fail after certain udev events
• LP: #2016858 (Curtin MP: 441347) - Fix issue where ntfsresize would not resize a NTFS partition that Ubiquity would due to differences in the arguments to ntfsresize.

Additional Bugfixes found in 23.04.1 and 23.04.2:

• ubuntu-desktop-installer/issues/1772 (PR: 1636) - Fix an issue where the EFI System Partition could be formatted without the installer user choosing to do so.
• LP: #2016901 (PR: 1660) - Fix issue where udev events could trigger queued partition changes being discarded, often leading to an install crash
• LP: #1983036 (PR: 1386) - Fix an issue affecting Ubuntu Desktop Installer where a partition which should have been formatted was not
• LP: #1989179 (PR: 1425) - Fix issue with "try again" when installing snaps to guarantee that snaps will be installed (was fixed previously, but not release noted before)
• LP: #1990239, LP: #1992531 (PR: 1429, PR: 1449) - Fix a crash issue at the end of the install due to unmount conflicts
• LP: #1998376 (PR: 1505) - Fix an issue where the wrong drivers were offered during install in Ubuntu Desktop Installer
• LP: #2004659 (PR: 1569) - Fix an issue where the online state would not be detected correctly for Desktop images, or any other case with Network Manager.
• LP: #2008271 (PR: 1578) - Fix an autoinstall issue where an incorrect keyboard layout value would fail late in the install. It now warns sooner with an Unknown keyboard ValueError.
• LP: #2012722 (Curtin MP: 439863) - Fix an issue where, when using Ventoy, a block probing failure would be raised early in the install.
• LP: #2015100 (Curtin MP: 440232) - Fix a disk lookup failure when the WWN value has a vendor extension
• LP: #2013079 (PR: 1618) - Fix Active Directory domain joins when using Ubuntu Desktop Installer. (Domain Joins not supported yet in Subiquity TUI)
• LP: #2015028 (PR: 1637) - Fix a crash that could occur with some keyboard layouts
• LP: #2015521 (PR: 1641) - Fix a crash that occur when attempting to create a logical partition.
• LP: #2002413 (PR: 1652) - Fix a crash when using Ubuntu Desktop Installer and configuring a swap partition.

New Contributors

• @EduardGomezEscandell made their first contribution in https://github.com/canonical/subiquity/pull/1434
• @rmounce made their first contribution in https://github.com/canonical/subiquity/pull/1579

Full Changelog: https://github.com/canonical/subiquity/compare/23.02.1...23.04.2

• Enable attaching the installed device to Ubuntu Pro, if the series being installed is a LTS
• LP: #2004040 - Set Systemd ShowStatus to off shortly before running Subiquity, to prevent service status messages from overwriting the screen
• LP: #2004609 - Update Curtin to pick up a fix for swap partition creation with MSDOS partition tables
• Update Probert to pick up a fix for reporting of swap partition sizes
• Update the logs for geoip to clarify that a failure to get geoip information is not fatal

• Add support for proxy-aware fetching of SSH keys
• LP: #1927103 - Fix issue in autoinstall with handoff of Swap value to Curtin
• Fix handling for when sshd is not present on the installation system
• LP: #1991413, LP: #1991929, LP: #1989977 - fix issues around usage of -1 in Autoinstall for assigning the rest of the disk to the last partition
• LP: #1999173 - Fix issue with deletion of an existing RAID
• LP: #1975860, https://github.com/canonical/probert/issues/125 - Update probert to fix handling of an inactive md raid
• LP: #1997920 - Update curtin to fix wiping of LUKS headers
• LP: #1892494 - Update curtin to handle the deprecation of apt-key, curtin now implements the policy suggested at https://blog.jak-linux.org/2021/06/20/migrating-away-apt-key/

• use_gap - The first available use case for Subiquity of Storage Version 2, a feature set exposing the ability to edit existing partition tables and resize disks, is the feature use_gap in autoinstall. use_gap allows for a guided install into existing free space without reformatting the entire disk. Example:

  # requires the subiquity-storage-version=2 kernel command line feature flag
  storage:
    layout:
      name: direct
      mode: use_gap
  

  In this scenario, a non-LVM guided install will be done into the largest free space. For more information on storage version 2, please see the Curtin configuration documentation.
• Better default keyboard layout and variant - the language chosen on the first screen of the Subiquity UI changes which keyboard layout and variant are suggested.
• Starting with Ubuntu Kinetic, cloud-init is configured in the installed system to disable itself after the first-boot configuration is complete.
• Integrate with cloud-init clean - several install artifacts are registered with cloud-init's clean functionality.
• Adjusted some UI colors with a goal of improved accessibility.
• LP: #1982903 - Fix 'Raid' object has no attribute 'path' error related to VROC
• LP: #1948823 - In the case where multiple copies of autoinstall data have been supplied, only one will be used. The precedence in that decision is
  1. /autoinstall.yaml (requires repacking the installer squashfs)
  2. Autoinstall provided by way of cloud-init
  3. /cdrom/autoinstall.yaml, which is where the autoinstall file will be located if placed in the root of the install ISO.
• LP: #1983265 - fix issue where static network configuration would produce a netplan config that used deprecated elements
• LP: #1987341 - add more users to the reserved user list to reduce cases where the user cannot be created
• LP: #1988407 - fix crash when editing logical volume on the confirmation screen
• LP: #1986674 - fix issue with Snap screen being shown despite being skipped. Interacting with the screen in this scenario "restarts" the install from the language screen.
• LP: #1989193, LP: #1989198 - Fix several failures related to the Snap screen and cancellation of retrieving the list of snaps
• LP: #1980584 - fix failure related to applying the apt configuration before the install source has been chosen
• LP: #1989353 - fix issue where the kernel package or flavor chosen in autoinstall was not respected
• LP: #1894009 - fix issue with the wrong keyboard layout in initramfs
• LP: #1992541 - fix bad partition placement for some disk sizes with guided LVM
• LP: #1992531, LP: #1992977 - fix crashes related to cleaning up mounts after running ubuntu-drivers
• LP: #1993257 - fix UI crash when reaching the custom disk layout screen if a storage probing operation is ongoing
• LP: #1987236 - fix issue in Curtin with s390x configurations that have nothing for chzdev to configure
• LP: #1980008 - fix issue in Curtin where cryptsetup wasn't always being installed on systems using it
• LP: #1992990 - fix issue in Curtin on relevant architectures where flash-kernel was not running due to the install happening in a chroot
• LP: #1992115 - fix issue in Probert that resulted in a crash related to /dev/loop0 not being found, as may be seen when performing a netboot install.