Bootable RPi4 / RPi3 image with 64-bit kernel, 32-bit Raspbian Buster host OS, 64-bit Debian Buster guest OS in nspawn container
31 Oct 2020: sadly, due to legal obligations arising from a recent change in my 'real world' job, I must announce I am standing down as maintainer of this project with immediate effect. For the meantime, I will leave the repo up (for historical interest, and since the Debian package is still in the official Raspbian repos); however, there will be no further updates to the underlying OS images etc., nor will I be accepting / actioning further pull requests or bug reports from this point. Email requests for support will also have to be politely declined, so, please treat this as an effective EOL notice.
For further details, please see my post here.
With sincere apologies, sakaki ><
Thanks to ShiftPlusOne, as of 2 March 2020 raspbian-nspawn-64 can be installed from the official Raspbian repo, just like any other regular package, using apt - so no need for a custom bootable image! For more details, please see this post, but to install, simply issue:
pi@raspberrypi:~ $ sudo apt-get update && sudo apt-get install -y raspbian-nspawn-64
As such, this v1.4.8 image is essentially just the 2020-02-13 with-desktop Buster release of Raspbian (available here), into which this raspbian-nspawn-64 package has been pre-installed, via the techniques shown in this post, and on which the official 64-bit kernel has been activated (by setting arm_64bit=1 in /boot/config.txt). The rootfs has also been expanded slightly from the original (to 5GiB), but otherwise, the system is in a 'factory fresh' state (so e.g. the root partition will still autoexpand, the piwiz setup wizard will still run on first boot, etc.). It is provided (given a deb now exists) as a convenience only.
Changes in this release (wrt v1.3.0):
Menu commands like System Tools→Run 64-bit Program... will now display an error dialog if attempted while running under a 32-bit kernel (they used to just fail silently).
CLI-commands (ds64-runner etc.) will not use zenity to display dialog boxes if this package is not installed (or the GUI isn't running), falling back to console output in this case.
The underlying 64-bit Debian Buster image deb (debian-buster-64) has been updated to a version debootstrapped on 19 Feb 2020.
Package metadata retained in the debian-buster-64 image this time (since a very common issue has been people not remembering to sudo apt-get update in the 64-bit shell, before trying to install new packages there — this way, while you may still need to update if it has become stale wrt certain packages, at least there's a helpful prompt to tell you that).
Added a mechanism to automatically bind mount 'shadow' copies of top-level dot-files and dot-directories for regular users, where an appropriate 'doppelganger' is provided. Specifically, for any (normal) file or directory ~/.foo, if a 'shadow' ~/.foo-debian-buster-64 is present, this will be automatically be bind mounted over the original inside the container only. No graphical editor for this is currently provided; however, it does allow for easy solution of the following (e.g.):
To work around the issue where e.g. chromium has different format ~/.config/... entries in the 32-bit host and 64-bit container, create the directory ~/.config-debian-buster-64 - this will then be used as ~/.config within the 64-bit container context only. (NB - the automatic bind mounts only happen at container startup, so you may need to reboot your system, or use System Tools→Stop 64-bit Container followed by System Tools→Start 64-bit Container to have your change 'take'.)
To have a separate 32-bit and 64-bit bash history, create the file ~/.bash_history-debian-buster-64 - this will then be used as ~/.bash_history within the 64-bit container context only. (Again, changes are applied at container startup.)
There are a number of regressions affecting 64-bit chromium v79 and v80 when run in a container (or when using a remote X server); these can make the browser window appear with all content 'whited out'. For details, please see e.g. chromium bug 1048186 and bug 1035803.
However, for chromium v79 at least (the current Debian Buster aarch64 version at the time of writing) you can install a working version of the browser via the following small workaround.
First, open a container shell, and if you haven't already, install 64-bit chromium:
pi@raspberrypi:~ $ ds64-shell
pi@debian-buster-64:~ $ sudo apt-get update
pi@debian-buster-64:~ $ sudo apt-get install -y chromium
Once complete, ensure (per the above bug reports) that the --ignore-gpu-blacklist option will always be given at browser startup. Issue:
pi@debian-buster-64:~ $ sudo nano -w /etc/chromium.d/ignore-gpu-blacklist
and place in that file:
# Workaround rendering failure on RPi, see bug #1048186 and bug #1035803
export CHROMIUM_FLAGS="$CHROMIUM_FLAGS --ignore-gpu-blacklist"
Save, and exit nano. You should now find the browser starts up correctly!
NB, to avoid conflict with the 32-bit
chromiumon the host desktop, you may also wish to set up a 'shadow'~/.configdirectory, as described above (although this is not mandatory).
Placeholder release.
NB: for the time being, please use the v1.3.0 release if you want a bootable image; this (v1.4.7) release has been created primarily for deb packaging purposes.
Placeholder release with added manpages, and slightly modified install script.
NB: for the time being, please use the previous (v1.3.0) release if you want a bootable image; this (v1.3.1) release has been created primarily for deb packaging purposes.
Changes in this release:
The host image has been updated to Raspbian Buster (specifically, the 10 July 2019 Raspbian Buster with desktop). This (together with the official 64-bit kernel, see next point) allows it to be used in 64-bit mode on the RPi4 B now, as well as (in 64-bit mode) on the RPi3 B and B+ (which remain supported). As shipped, the image has not yet been first-time booted (although the various reflector services etc. have also been installed onto it, using the technique described here, and some additional deps have been installed (via # apt-get update && apt-get -y upgrade && apt-get install -y debootstrap pulseaudio zenity systemd-container file locales sudo libpam-systemd dbus-user-session), so the user will get the 'factory fresh' Raspbian experience (root partition / filesystem auto-expansion etc.)
Also using this technique, a guest Debian Buster image has been created at /var/lib/machines/debian-buster-64, using the command # debootstrap --arch=arm64 --include=systemd-container,file,locales,pulseaudio,zenity,firefox-esr,x11-apps,dbus-user-session,libpam-systemd buster,sudo /var/lib/machines/debian-buster-64 https://deb.debian.org/debian/. The guest image has not been first-time booted either, as shipped. All non-vanilla adaptations to it are made by the host-side init-container service / script, when the system is started.
For convenience, a tarball of the resulting guest image has been provided as part of this release (debian-buster-64.tar.xz, the sha256sum of which is 5ac118c391bd7ffb36f717273b96e53d64f06b5b9cff66ecef552b6486470868). NB:, this tarball is not required to use the main image (as it has already been installed on there), but may be of interest to those looking to e.g. package the system.
Switched to using the newly-released official 64-bit kernel, for both the RPi4 and RPi3. Since, following upstream's recommendations, the new kernel was installed using # rpi-update, as a side-effect of the switch the other boot firmware on the image has been freshened also.
The installed kernel release name is 4.19.69-v8+ (and should be safely updatable in future, using rpi-update).
Added a temporary host-side service to enable rngd (via the rng-tools service) only on the RPi4 (the current official 64-bit kernel does not seem to enable /dev/hwrng properly on the RPi3 yet).
For avoidance of doubt, the image has had arm64_bit=1 and dtoverlay=vc4-fkms-v3d set in its /boot/config.txt.
Added a host-side Xsession.d rule to create a copy of the user's .Xauthority file that has FamilyWild authentication set. This allows it to be used even from within the guest (which has a different hostname). Modified ds64-run and ds64-shell to use this tweaked .Xauthority-allhosts file. This fix also allows sudo to be used with guest GUI applications, where required (this didn't work in the previous release).
Because the Mesa / libgl versions on the current Debian Buster don't seem to be able to work with the RPi 4/3's vc* GPU correctly (even if /dev/dri is bind-mounted, via an entry in /etc/systemd/nspawn/debian-buster-64.nspawn), set LIBGL_ALWAYS_SOFTWARE=1 by default on the guest (to ensure that accelerated apps, such as Chromium, can at least display), and ensured this is propagated by sudo. This behaviour is parameterized via an entry (GUEST_LIBGL_ALWAYS_SOFTWARE) in /etc/ds64.conf (host-side), so you can easily turn it off again if you wish.
Added the pulseaudio "glitchy playback" fix (by setting tsched=0 in /etc/pulse/default.pa, on both host and guest side). Credit: Darksky.
Added USER, LANG, NO_AT_BRIDGE and DBUS_SESSION_BUS_ADDRESS to the default environment variables passed by ds64-run and ds64-shell, and expanded the default PATH passed by both to include /usr/local/sbin, /usr/sbin and /sbin.
Updated the install.sh script.
Other minor stability fixes.
Changes in this release:
apt-get etc. (the upstream autobuild is here). The particular kernel version shipped on the image is 4.19.34-v8-43958a67195d-bis+./etc/default/locale on the guest (via this path unit), and when seen triggers a script to update a matching /etc/default/host-locale file within the guest's filesystem. There, a counterpart path unit, service and script act on any changes (whenever the guest is running), to bring the guest locale in line. If the necessary locale is not present in the guest, it will be compiled automatically (note that locales are never removed from /etc/locale.gen automatically, only added). The counterpart path unit may be disabled in the guest, if locale reflection is not desired./etc/timezone on the host (via this path unit), and when seen triggers a script to update a matching /etc/host-timezone file within the guest's filesystem. There, a counterpart path unit, service and script act on any changes (whenever the guest is running), to bring the guest time zone in line.init-container service on the host (unit, script) to prepare the guest image prior to its startup. This takes care of installing all context-specific units, scripts etc. into the guest (meaning that other than ensuring the latter has all necessary packages installed - which for avoidance of doubt the version on this image has - no custom prep is required). One of the services enabled by this in the guest is the container-init counterpart (unit, script) which performs some additional setup when the guest image is booted (hostname conformance, etc.)./usr/{s,}bin rather than /usr/local/bin etc.) A host image directory is now provided (here).$DESTDIR, and understands preinst, install, postinst, prerm, uninstall / purge, and postrm actions).sha256sum of the debian-buster-64.tar.xz image file (only required for packaging etc. - already installed on main image): 3a3c1433caf3a1f89fb624081fdb1799b7ed20376d28c327affc711858204595
Changes in this release:
Kernel upgraded to 4.14.97-v8-0448a1dbea0f-bis+. The bcm2835-unicam module has been included (but not loaded by default at boot).
Added a systemd path unit and triggered script (/usr/local/bin/reflect-apps), to watch for changes to /usr/share/applications within the container. When triggered, copies all .desktop files across to the host, prefixing their names to ensure they are unique, and also modifying their Exec= stanzas (with ds64-runner or ds64-shell -c, as appropriate) to allow direct invocation from a 32-bit context. Desktop files without Type=Application are ignored, and any prior desktop files at the target location in the host, which have NoDisplay=true set, will not be overwritten. Also copies contents of /usr/share/{icons,pixmaps} from the guest into the host (at /usr/share/gdm/{icons,pixmaps}), so (most) referenced icons will resolve, and reloads the (host's) main menu, so these changes are picked up.
The net effect of this script is that when a new 64-bit application is installed in the guest, a menu entry to launch it (so it can play audio, display on the desktop etc.) should auto-magically get added to the host's main menu, complete with icon. Such items will also be automatically removed should the package subsequently be uninstalled. Please note that the script will wait for all apt-get, apt and dpkg processes to complete before making modifications.
Added a systemd path unit and triggered script (/usr/local/bin/reflect-passwd), to watch for changes to /etc/{passwd,shadow,group,gshadow) within the host. When triggered, reflects user data (including hashed passwords) in the 1000 <= uid < 1100 range into the container, ensuring that the primary group is also present. Removes clashing users or groups from the guest, and ensures that reflected users are members of groups cited in $USE_GROUP (see the script for details), iff such groups are present on the guest.
The net effect of this script is that if you change password or create a new user on the host, it should auto-magically be carried over into the guest as well. No equivalent propagation of changes from guest to host is provided.
Made all members of the sudo group (in both host and guest) eligible for password-free invocation of all commands (not just the pi user).
Migrated 64-bit utility .desktop files from pi's ~/.local/share/applications directory, to /usr/local/share/applications, as they may now be used by other regular users, not just pi.
Mapped all of /home into the container, not just /home/pi.
Removed hardcoded 1000 uid and gid from /usr/local/bin/ds64-run and /usr/local/sbin/ds64-shell.
Allowed passing of parameters to ds64-shell (so it can be used to e.g. launch terminal-based 64-bit apps from the guest, such as htop etc.)
Various other minor clean-ups.
sha256sum of the (xz-compressed) image file: 555ace66eb82299fe358711c4c69e98bd1b641a99abd2e78ca69d66f53ce348b.
Initial public release.
Configuration:
sha256sum of the (xz-compressed) image file: 6d74fea112d96fccd21616ab6fc31465a8576200ab87881c7a4ff856d32b384d.