Cryptsetup sprint report
The Cryptsetup team – consisting of Guilhem and Jonas – met on June 15 to 17 in order to work on the Debian cryptsetup packages. We ended up working three days (and nights) on the packages, refactored the whole initramfs integration, the SysVinit init scripts and the package build process and discussed numerous potential improvements as well as new features. The whole sprint was great fun and we enjoyed a lot sitting next to each other, being able to discuss design questions and implementation details in person instead of using clunky internet communication means. Besides, we had very nice and interesting chats, contacted other Debian folks from the Frankfurt area and met with jfs on Friday evening.
Splitting cryptsetup into cryptsetup-run and cryptsetup-initramfs
First we split the cryptsetup initramfs integration into a separate package
cryptsetup-initramfs. The package that contains other Debian specific
features like SysVinit scripts, keyscripts, etc. now is called
cryptsetup-run and cryptsetup itself is a mere metapackage depending on
both split off packages. So from now on, people can install cryptsetup-run
if they don't need the cryptsetup initramfs integration. Once Buster is
released we intend to rename cryptsetup-run to cryptsetup, which then
will no longer have a strict dependency on cryptsetup-initramfs. This
transition over two releases is necessary to avoid unexpected breakage on
(dist-)upgrades. Meanwhile cryptsetup-initramfs ships a hook that upon
generation of a new initramfs image detects which devices need to be
unlocked early in the boot process and, in case it didn't find any, suggests
the user to remove the package.
The package split allows us to define more fine-grained dependencies: since
there are valid use case for wanting the cryptsetup binaries scripts but not
the initramfs integration (in particular, on systems without encrypted root
device), cryptsetup ≤2:2.0.2-1 was merely recommending initramfs-tools
and busybox, while cryptsetup-initramfs now has hard dependencies on
these packages.
We also updated the packages to latest upstream release and uploaded 2:2.0.3-1 on Friday shortly before 15:00 UTC. Due to the cryptsetup → cryptsetup-{run,initramfs} package split we hit the NEW queue, and it was manually approved by an ftpmaster… a mere 2h later. Kudos to them! That allowed us to continue with subsequent uploads during the following days, which was beyond our expectations for this sprint :-)
Extensive refactoring work
Afterwards we started working on and merging some heavy refactoring commits
that touched almost all parts of the packages. First was a refactoring of
the whole cryptsetup initramfs implementation that downsized both the
cryptroot hook and script dramatically (less than half the size they were
before). The logic to detect crypto disks was changed from parsing
/etc/fstab to /proc/mounts and now the sysfs(5) block hierarchy is used to
detect dm-crypt device dependencies. A lot of code duplication between the
initramfs script and the SysVinit init script was removed by outsourcing
common functions into a shared shell functions include file that is sourced
by initramfs and SysVinit scripts. To complete the package refactoring, we
also overhauled the build process by migrating it to the latest Debhelper 11
style. debian/rules as well was downsized to less than half the size and
as an extra benefit we now run the upstream build-time testsuite during the
package build process.
Some git statistics speak more than a thousand words:
$ git --no-pager diff --ignore-space-change --shortstat debian/2%2.0.2-1..debian/2%2.0.3-2 -- ./debian/
92 files changed, 2247 insertions(+), 3180 deletions(-)
$ find ./debian -type f \! -path ./debian/changelog -print0 | xargs -r0 cat | wc -l
7342
$ find ./debian -type f \! -path ./debian/changelog -printf x | wc -c
106
On CVE-2016-4484
Since 2:1.7.3-2, our initramfs boot script went to sleep for a full minute
when the number of failed unlocking attempts exceeds the configured value
(tries crypttab(5) option, which defaults to 3). This was added in order
to defeat local brute force attacks, and mitigate one aspect of
CVE-2016-4484;
back then Jonas wrote a blog post to cover that story. Starting with
2:2.0.3-2 we changed this behavior and the script will now sleep for one
second after each unsuccessful unlocking attempt. The new value should
provide better user experience while still offering protection against local
brute force attacks for very fast password hashing functions. The other
aspect mentioned in the security advisory — namely the fact that the initramfs
boot process drops to a root (rescue/debug) shell after the user fails to
unlock the root device too many times — was not addressed at the time, and
still isn't. initramfs-tools has a boot parameter panic=<sec> to disable the
debug shell, and while setting this is beyond the scope of cryptsetup, we're
planing to ask the initramfs-tools maintainers to change the default. (Of
course setting panic=<sec> alone doesn't gain much, and one would need to
lock down the full boot chain, including BIOS and boot loader.)
New features (work started)
Apart from the refactoring work we started/continued work on several new features:
- We started to integrate luksSuspend support into system suspend. The idea
is to luksSuspend all dm-crypt devices before suspending the machine in
order to protect the storage in suspend mode. In theory, this seemed as
simple as creating a minimal chroot in ramfs with the tools required to
unlock (luksResume) the disks after machine resume, running luksSuspend from
that chroot, putting the machine into suspend mode and running
luksResumeafter it got resumed. Unfortunately it turned out to be way more complicated due to unpredictable race conditions betweenluksSuspendand machine suspend. So we ended up spending quite some time on debugging (and understanding) the issue. In the end it seems like the final sync() before machine suspend ( https://lwn.net/Articles/582648/ ) causes races in some cases as the dm-crypt device to be synced to is already luksSuspended. We ended up sending a request for help to the dm-crypt mailinglist but unfortunately so far didn't get a helpful response yet. - In order to get internationalization support for the messages and password prompts in the initramfs scripts, we patched gettext and locale support into initramfs-tools.
- We started some preliminary work on adding beep support to the cryptsetup initramfs and sysVinit scripts for better accessibility support.
The above features are not available in the current Debian package yet, but we hope they will be included in a future release.
Bugs and Documentation
We also squashed quite some longstanding bugs and improved the crypttab(5) documentation. In total, we squashed 18 bugs during the sprint, the oldest one being from June 2013.
On the need for better QA
In addition to the many crypttab(5) options we also support a huge variety of block device stacks, such as LUKS-LVM2-MD combined in all ways one can possibly imagine. And that's a Debian addition hence something we, the cryptsetup package maintainers, have to develop and maintain ourselves. The many possibilities imply corner cases (it's not a surprise that complex or unusual setups can break in subtle ways) which motivated us to completely refactor the Debian-specific code, so it becomes easier to maintain.
While our final upload squashed 18 bugs, it also introduced new ones. In particular 2 rather serious regressions which fell through our tests. We have thorough tests for the most usual setups, as well as for some complex stacks we hand-crafted in order to detect corner cases, but this approach doesn't scale to covering the full spectrum of user setups: even with minimal sid installations the disk images would just take far too much space! Ideally we would have a automated test-suite, each test deploying a new transient sid VM with a particular setup. As the current and past regressions show, that's a beyond-the-scenes area we should work on. (In fact that's an effort we started already, but didn't touch during the sprint due to lack of time.)
More to come
There's some more things on our list that we didn't find time to work on. Apart from the unfinished new features we mentioned above, that's mainly the LUKS nuke feature that Kali Linux ships and the lack of keyscripts support to crypttab(5) in systemd.
Conclusion
In our eyes, the sprint was both a great success and great fun. We definitely want to repeat it anytime soon in order to further work on the open tasks and further improve the Debian cryptsetup package. There's still plenty of work to be done. We thank the Debian project and its generous donors for funding Guilhem's travel expenses.
Guilhem and Jonas, June 25th 2018