Debian LTS report for June 2019

This month I was allocated 17 hours. I also had 1.75 hours left over from May, which makes a total of 18.75 hours. I spent 16.75h of them on the following issues, which means I again carry over 2h to the next month.

Posted Mon 01 Jul 2019 12:59:09 PM UTC Tags:

Debian LTS report for May 2019

This month I was allocated 17 hours. I spent 15.25 hours on the following issues:

  • DLA 1766-1: OpenPGP signature spoofing in evolution. On this issue I actually spent way more time than expected during April. I took over some of the remaining hours to May.
  • DLA 1778-1: Several vulnerabilities in symfony, a PHP web application framework.
  • DLA 1791-1: Several vulnerabilities in drupal7, a PHP web site platform.
Posted Tue 04 Jun 2019 05:24:31 PM UTC Tags:

Debian LTS report for April 2019

After a longer break (~two years) I again took part in the funded Debian LTS project in April 2019.

I was allocated 14 hours and spent all of them (and even a bit more) on the following two issues:

  • DLA 1748-1: Several security fixes for apache2
  • DLA 1766-1: OpenPGP signature spoofing in evolution
Posted Fri 26 Apr 2019 09:20:47 PM UTC Tags:

Cryptsetup sprint report

The Cryptsetup team – consisting of Guilhem and Jonas – met on June 15 to 17 in order to work on the Debian cryptsetup packages. We ended up working three days (and nights) on the packages, refactored the whole initramfs integration, the SysVinit init scripts and the package build process and discussed numerous potential improvements as well as new features. The whole sprint was great fun and we enjoyed a lot sitting next to each other, being able to discuss design questions and implementation details in person instead of using clunky internet communication means. Besides, we had very nice and interesting chats, contacted other Debian folks from the Frankfurt area and met with jfs on Friday evening.

Splitting cryptsetup into cryptsetup-run and cryptsetup-initramfs

First we split the cryptsetup initramfs integration into a separate package cryptsetup-initramfs. The package that contains other Debian specific features like SysVinit scripts, keyscripts, etc. now is called cryptsetup-run and cryptsetup itself is a mere metapackage depending on both split off packages. So from now on, people can install cryptsetup-run if they don't need the cryptsetup initramfs integration. Once Buster is released we intend to rename cryptsetup-run to cryptsetup, which then will no longer have a strict dependency on cryptsetup-initramfs. This transition over two releases is necessary to avoid unexpected breakage on (dist-)upgrades. Meanwhile cryptsetup-initramfs ships a hook that upon generation of a new initramfs image detects which devices need to be unlocked early in the boot process and, in case it didn't find any, suggests the user to remove the package.

The package split allows us to define more fine-grained dependencies: since there are valid use case for wanting the cryptsetup binaries scripts but not the initramfs integration (in particular, on systems without encrypted root device), cryptsetup ≤2:2.0.2-1 was merely recommending initramfs-tools and busybox, while cryptsetup-initramfs now has hard dependencies on these packages.

We also updated the packages to latest upstream release and uploaded 2:2.0.3-1 on Friday shortly before 15:00 UTC. Due to the cryptsetup → cryptsetup-{run,initramfs} package split we hit the NEW queue, and it was manually approved by an ftpmaster… a mere 2h later. Kudos to them! That allowed us to continue with subsequent uploads during the following days, which was beyond our expectations for this sprint :-)

Extensive refactoring work

Afterwards we started working on and merging some heavy refactoring commits that touched almost all parts of the packages. First was a refactoring of the whole cryptsetup initramfs implementation that downsized both the cryptroot hook and script dramatically (less than half the size they were before). The logic to detect crypto disks was changed from parsing /etc/fstab to /proc/mounts and now the sysfs(5) block hierarchy is used to detect dm-crypt device dependencies. A lot of code duplication between the initramfs script and the SysVinit init script was removed by outsourcing common functions into a shared shell functions include file that is sourced by initramfs and SysVinit scripts. To complete the package refactoring, we also overhauled the build process by migrating it to the latest Debhelper 11 style. debian/rules as well was downsized to less than half the size and as an extra benefit we now run the upstream build-time testsuite during the package build process.

Some git statistics speak more than a thousand words:

$ git --no-pager diff --ignore-space-change --shortstat debian/2%2.0.2-1..debian/2%2.0.3-2 -- ./debian/
 92 files changed, 2247 insertions(+), 3180 deletions(-)
$ find ./debian -type f \! -path ./debian/changelog -print0 | xargs -r0 cat | wc -l
7342
$ find ./debian -type f \! -path ./debian/changelog -printf x | wc -c
106

On CVE-2016-4484

Since 2:1.7.3-2, our initramfs boot script went to sleep for a full minute when the number of failed unlocking attempts exceeds the configured value (tries crypttab(5) option, which defaults to 3). This was added in order to defeat local brute force attacks, and mitigate one aspect of CVE-2016-4484; back then Jonas wrote a blog post to cover that story. Starting with 2:2.0.3-2 we changed this behavior and the script will now sleep for one second after each unsuccessful unlocking attempt. The new value should provide better user experience while still offering protection against local brute force attacks for very fast password hashing functions. The other aspect mentioned in the security advisory — namely the fact that the initramfs boot process drops to a root (rescue/debug) shell after the user fails to unlock the root device too many times — was not addressed at the time, and still isn't. initramfs-tools has a boot parameter panic=<sec> to disable the debug shell, and while setting this is beyond the scope of cryptsetup, we're planing to ask the initramfs-tools maintainers to change the default. (Of course setting panic=<sec> alone doesn't gain much, and one would need to lock down the full boot chain, including BIOS and boot loader.)

New features (work started)

Apart from the refactoring work we started/continued work on several new features:

  • We started to integrate luksSuspend support into system suspend. The idea is to luksSuspend all dm-crypt devices before suspending the machine in order to protect the storage in suspend mode. In theory, this seemed as simple as creating a minimal chroot in ramfs with the tools required to unlock (luksResume) the disks after machine resume, running luksSuspend from that chroot, putting the machine into suspend mode and running luksResume after it got resumed. Unfortunately it turned out to be way more complicated due to unpredictable race conditions between luksSuspend and machine suspend. So we ended up spending quite some time on debugging (and understanding) the issue. In the end it seems like the final sync() before machine suspend ( https://lwn.net/Articles/582648/ ) causes races in some cases as the dm-crypt device to be synced to is already luksSuspended. We ended up sending a request for help to the dm-crypt mailinglist but unfortunately so far didn't get a helpful response yet.
  • In order to get internationalization support for the messages and password prompts in the initramfs scripts, we patched gettext and locale support into initramfs-tools.
  • We started some preliminary work on adding beep support to the cryptsetup initramfs and sysVinit scripts for better accessibility support.

The above features are not available in the current Debian package yet, but we hope they will be included in a future release.

Bugs and Documentation

We also squashed quite some longstanding bugs and improved the crypttab(5) documentation. In total, we squashed 18 bugs during the sprint, the oldest one being from June 2013.

On the need for better QA

In addition to the many crypttab(5) options we also support a huge variety of block device stacks, such as LUKS-LVM2-MD combined in all ways one can possibly imagine. And that's a Debian addition hence something we, the cryptsetup package maintainers, have to develop and maintain ourselves. The many possibilities imply corner cases (it's not a surprise that complex or unusual setups can break in subtle ways) which motivated us to completely refactor the Debian-specific code, so it becomes easier to maintain.

While our final upload squashed 18 bugs, it also introduced new ones. In particular 2 rather serious regressions which fell through our tests. We have thorough tests for the most usual setups, as well as for some complex stacks we hand-crafted in order to detect corner cases, but this approach doesn't scale to covering the full spectrum of user setups: even with minimal sid installations the disk images would just take far too much space! Ideally we would have a automated test-suite, each test deploying a new transient sid VM with a particular setup. As the current and past regressions show, that's a beyond-the-scenes area we should work on. (In fact that's an effort we started already, but didn't touch during the sprint due to lack of time.)

More to come

There's some more things on our list that we didn't find time to work on. Apart from the unfinished new features we mentioned above, that's mainly the LUKS nuke feature that Kali Linux ships and the lack of keyscripts support to crypttab(5) in systemd.

Conclusion

In our eyes, the sprint was both a great success and great fun. We definitely want to repeat it anytime soon in order to further work on the open tasks and further improve the Debian cryptsetup package. There's still plenty of work to be done. We thank the Debian project and its generous donors for funding Guilhem's travel expenses.

Guilhem and Jonas, June 25th 2018

Posted Tue 26 Jun 2018 02:26:42 PM UTC Tags:

Debian LTS report for March 2017

March 2017 was my seventh month as a Debian LTS team member. I was allocated 14,75 hours and spent 11,25 of them on the following tasks:

  • DLA 836-2: Regression update for munin
  • DLA 869-1: Several security fixes for cgiemail
  • tested packaged prepared by other LTS team members against regressions
  • libical: bug triaging, testing reproducers
  • putty: tested reproducers, backported patches (not finished yet)
Posted Mon 03 Apr 2017 01:37:33 PM UTC Tags:

Debian LTS report for February 2017

February 2017 was my sixth month as a Debian LTS team member. I was allocated 5 hours and had 9,75 hours left over from January 2017. This makes a total of 14,75 hours. I spent all of them doing the following:

  • DLA 831-1: Fix buffer overflows in gtk-vnc
  • Reviewed the apache2 2.2.22-13+deb7u8 upload, improved the patches
  • Reviewed CVE-2017-5666 (mp3splt)
  • DLA 836-1: Fix command injection vulnerability in munin cgi script
Posted Sat 25 Feb 2017 05:22:01 PM UTC Tags:

Debian LTS report for January 2017

January 2017 was my fifth month as a Debian LTS team member. I was allocated 12 hours and had 6,75 hours left over from December 2016. This makes a total of 18,75 hours. Unfortunately I found less time than expected to work on Debian LTS in January. In total, I spent 9 hours on the following security updates:

  • DLA 787-1: XSS protection via Content Security Policy for otrs2
  • DLA 788-1: fix vulnerability in pdns-recursor by dropping illegitimate long querys
  • DLA 798-1: fix multiple vulnerabilities in pdns
Posted Fri 10 Feb 2017 04:07:57 PM UTC Tags:

Debian LTS report for December 2016

December 2016 was my fourth month as a Debian LTS team member. I was allocated 12 hours. Unfortunately I turned out to have way less time for Debian and LTS work than expected, so I only spent 5,25 hours of them for the following tasks:

  • DLA 732-1: backported CSRF protection to monit 1:5.4-2+deb7u1
  • DLA 732-2: fix a regression introduced in last monit security update
  • DLA 732-3: fix another regression introduced in monit security update
  • Nagios3: port 3.4.1-3+deb7u2 and 3.4.1-3+deb7u3 updates to wheezy-backports
  • DLA-760-1: fix two reflected XSS vulnerabilities in spip
Posted Sun 08 Jan 2017 10:18:56 AM UTC Tags:

On CVE-2016-4484, a (security)? bug in the cryptsetup initramfs integration

On November 4, I was made aware of a security vulnerability in the integration of cryptsetup into initramfs. The vulnerability was discovered by security researchers Hector Marco and Ismael Ripoll of CyberSecurity UPV Research Group and got CVE-2016-4484 assigned.

In this post I'll try to reflect a bit on

What CVE-2016-4484 is all about

Basically, the vulnerability is about two separate but related issues:

1. Initramfs rescue shell considered harmful

The main topic that Hector Marco and Ismael Ripoll address in their publication is that Debian exits into a rescue shell in case of failure during initramfs, and that this can be triggered by entering a wrong password ~93 times in a row.

Indeed the Debian initramfs implementation as provided by initramfs-tools exits into a rescue shell (usually a busybox shell) after a defined amount of failed attempts to make the root filesystem available. The loop in question is in local_device_setup() at the local initramfs script

In general, this behaviour is considered as a feature: if the root device hasn't shown up after 30 rounds, the rescue shell is spawned to provide the local user/admin a way to debug and fix things herself.

Hector Marco and Ismael Ripoll argue that in special environments, e.g. on public computers with password protected BIOS/UEFI and bootloader, this opens an attack vector and needs to be regarded as a security vulnerability:

It is common to assume that once the attacker has physical access to the computer, the game is over. The attackers can do whatever they want. And although this was true 30 years ago, today it is not.

There are many "levels" of physical access. [...]

In order to protect the computer in these scenarios: the BIOS/UEFI has one or two passwords to protect the booting or the configuration menu; the GRUB also has the possibility to use multiple passwords to protect unauthorized operations.

And in the case of an encrypted system, the initrd shall block the maximum number of password trials and prevent the access to the computer in that case.

While Hector and Ismael have a valid point in that the rescue shell might open an additional attack vector in special setups, this is not true for the vast majority of Debian systems out there: in most cases a local attacker can alter the boot order, replace or add boot devices, modify boot options in the (GNU GRUB) bootloader menu or modify/replace arbitrary hardware parts.

The required scenario to make the initramfs rescue shell an additional attack vector is indeed very special: locked down hardware, password protected BIOS and bootloader but still local keyboard (or serial console) access are required at least.

Hector and Ismael argue that the default should be changed for enhanced security:

[...] But then Linux is used in more hostile environments, this helpful (but naive) recovery services shall not be the default option.

For the reasons explained about, I tend to disagree to Hectors and Ismaels opinion here. And after discussing this topic with several people I find my opinion reconfirmed: the Debian Security Team disputes the security impact of the issue and others agree.

But leaving the disputable opinion on a sane default aside, I don't think that the cryptsetup package is the right place to change the default, if at all. If you want added security by a locked down initramfs (i.e. no rescue shell spawned), then at least the bootloader (GNU GRUB) needs to be locked down by default as well.

To make it clear: if one wants to lock down the boot process, bootloader and initramfs should be locked down together. And the right place to do this would be the configurable behaviour of grub-mkconfig. Here, one can set a password for GRUB and the boot parameter 'panic=1' which disables the spawning of a rescue shell in initramfs.

But as mentioned, I don't agree that this would be sane defaults. The vast majority of Debian systems out there don't have any security added by locked down bootloader and initramfs and the benefit of a rescue shell for debugging purposes clearly outrivals the minor security impact in my opinion.

For the few setups which require the added security of a locked down bootloader and initramfs, we already have the relevant options documented in the Securing Debian Manual:

After discussing the topic with initramfs-tools maintainers today, Guilhem and me (the cryptsetup maintainers) finally decided to not change any defaults and just add a 'sleep 60' after the maximum allowed attempts were reached.

2. tries=n option ignored, local brute-force slightly cheaper

Apart from the issue of a rescue shell being spawned, Hector and Ismael also discovered a programming bug in the cryptsetup initramfs integration. This bug in the cryptroot initramfs local-top script allowed endless retries of passphrase input, ignoring the tries=n option of crypttab (and the default of 3). As a result, theoretically unlimited attempts to unlock encrypted disks were possible when processed during initramfs stage. The attack vector here was that local brute-force attacks are a bit cheaper. Instead of having to reboot after max tries were reached, one could go on trying passwords.

Even though efficient brute-force attacks are mitigated by the PBKDF2 implementation in cryptsetup, this clearly is a real bug.

The reason for the bug was twofold:

  • First, the condition in setup_mapping() responsible for making the function fail when the maximum amount of allowed attempts is reached, was never met:

    setup_mapping()
    {
      [...]
      # Try to get a satisfactory password $crypttries times
      count=0                              
    while [ $crypttries -le 0 ] || [ $count -lt $crypttries ]; do export CRYPTTAB_TRIED="$count" count=$(( $count + 1 )) [...] done if [ $crypttries -gt 0 ] && [ $count -gt $crypttries ]; then message "cryptsetup: maximum number of tries exceeded for $crypttarget" return 1 fi [...] }

    As one can see, the while loop stops when $count -lt $crypttries. Thus the second condition $count -gt $crypttries is never met. This can easily be fixed by decreasing $count by one in case of a successful unlock attempt along with changing the second condition to $count -ge $crypttries:

    setup_mapping()
    {
      [...]
      while [ $crypttries -le 0 ] || [ $count -lt $crypttries ]; do
          [...]
          # decrease $count by 1, apparently last try was successful.
          count=$(( $count - 1 ))
          [...]
      done
      if [ $crypttries -gt 0 ] && [ $count -ge $crypttries ]; then
          [...]
      fi
      [...]
    }
    

    Christian Lamparter already spotted this bug back in October 2011 and provided a (incomplete) patch, but back then I even managed to merge the patch in an improper way, making it even more useless: The patch by Christian forgot to decrease $count by one in case of a successful unlock attempt, resulting in warnings about maximum tries exceeded even for successful attemps in some circumstances. But instead of adding the decrease myself and keeping the (almost correct) condition $count -eq $crypttries for detection of exceeded maximum tries, I changed back the condition to the wrong original $count -gt $crypttries that again was never met. Apparently I didn't test the fix properly back then. I definitely should do better in future!

  • Second, back in December 2013, I added a cryptroot initramfs local-block script as suggested by Goswin von Brederlow in order to fix bug #678692. The purpose of the cryptroot initramfs local-block script is to invoke the cryptroot initramfs local-top script again and again in a loop. This is required to support complex block device stacks.

    In fact, the numberless options of stacked block devices are one of the biggest and most inglorious reasons that the cryptsetup initramfs integration scripts became so complex over the years. After all we need to support setups like rootfs on top of LVM with two separate encrypted PVs or rootfs on top of LVM on top of dm-crypt on top of MD raid.

    The problem with the local-block script is that exiting the setup_mapping() function merely triggers a new invocation of the very same function.

    The guys who discovered the bug suggested a simple and good solution to this bug: When maximum attempts are detected (by second condition from above), the script sleeps for 60 seconds. This mitigates the brute-force attack options for local attackers - even rebooting after max attempts should be faster.

About disclosure, wording and clickbaiting

I'm happy that Hector and Ismael brought up the topic and made their argument about the security impacts of an initramfs rescue shell, even though I have to admit that I was rather astonished about the fact that they got a CVE assigned.

Nevertheless I'm very happy that they informed the Security Teams of Debian and Ubuntu prior to publishing their findings, which put me in the loop in turn. Also Hector and Ismael were open and responsive when it came to discussing their proposed fixes.

But unfortunately the way they advertised their finding was not very helpful. They announced a speech about this topic at the DeepSec 2016 in Vienna with the headline Abusing LUKS to Hack the System.

Honestly, this headline is missleading - if not wrong - in several ways:

  • First, the whole issue is not about LUKS, neither is it about cryptsetup itself. It's about Debians integration of cryptsetup into the initramfs, which is a compeletely different story.
  • Second, the term hack the system suggests that an exploit to break into the system is revealed. This is not true. The device encryption is not endangered at all.
  • Third - as shown above - very special prerequisites need to be met in order to make the mere existance of a LUKS encrypted device the relevant fact to be able to spawn a rescue shell during initramfs.

Unfortunately, the way this issue was published lead to even worse articles in the tech news press. Topics like Major security hole found in Cryptsetup script for LUKS disk encryption or Linux Flaw allows Root Shell During Boot-Up for LUKS Disk-Encrypted Systems suggest that a major security vulnerabilty was revealed and that it compromised the protection that cryptsetup respective LUKS offer.

If these articles/news did anything at all, then it was causing damage to the cryptsetup project, which is not affected by the whole issue at all.

After the cat was out of the bag, Marco and Ismael aggreed that the way the news picked up the issue was suboptimal, but I cannot fight the feeling that the over-exaggeration was partly intended and that clickbaiting is taking place here. That's a bit sad.

Posted Tue 06 Dec 2016 02:21:10 PM UTC Tags:

Debian LTS report for November 2016

Noevember 2016 was my third month as a Debian LTS team member. I was allocated 11 hours and had 1,75 hours left from October. This makes a total of 12,75 hours. In November I spent all 12,75 hours (and even a bit more) preparing security updates for spip, memcached and monit.

In particular, the updates of spip and monit took a lot of time (each one more than six hours). The patches for both packages were horrible to backport as the affected codebase changed a lot between the Wheezy versions and current upstream versions. Still it was great fun and I learned a lot during the backporting work. Due to the intrusive nature of the patches, I also did much more extensive testing before uploading the packages, which took quite a bit of time as well.

Monit 5.4-2+deb7u1 is not uploaded to wheezy-security yet as I decided to ask for further review and testing on the debian-lts mailinglist first.

Below follows the list of items I worked on in November in the well known format:

  • DLA 695-1: several XSS, CSRF and code execution flaws fixed in spip 2.1.17-1+deb7u6
  • DLA 701-1: integer overflows, buffer over-read fixed in memcached 1.4.13-0.2+deb7u2
  • CVE-2016-7067: backported CSRF protection to monit 5.4-2+deb7u1
Posted Wed 30 Nov 2016 07:30:18 PM UTC Tags:

This blog is powered by ikiwiki.