jenkins.debian.net or what is Debian doing with all these ressources
Holger Levsen (h01ger)
about me
- Debian user since 1995
- Debian contributor since 2001
- DebConf organizer, founded the DebConf video team in 2005
- Debian developer since 2007
- Debian-Edu (Debian for Education), since 2003
- Debian-QA
- Freelancer at Profitbricks from 2012-2015
more about Debian QA and me
- https://piuparts.debian.org/ since 2009
- https://jenkins.debian.net/ since 2012
- https://reproducible.debian.net/ since 2014
- since 2015 funded by the Linux Foundation
Profitbricks ressources used by jenkins.debian.net
- FIXME: check those numbers
- 2 datacenters (KA + FRA)
- 15 machines
- 131 cores (20 Intel, 111 AMD)
- 500 GB RAM
- 3 TB HDD storage
- 2 TB SDD storage
- no static IP addresses
Ressources used by reproducible.debian.net, by architecture
- FIXME: check those numbers, page is out of place here
- 15 amd64 systems, sponsored by Profitbricks
- 4 i386 systems, sponsored by Profitbricks
- 22 armhf systems, sponsored by vagrant@d.o, Debian & other donations
- soon: 8 arm64 systems, sponsored by codethink.co.uk
The problem
- Can inspect the source code of free software for flaws
- But distributions provide binary/compiled packages
Can we trust this process?
- To get users, go after the developers
- Financial incentives to crack developer machines / build infrastructure
CVE-2002-0083
: Remote root exploit in OpenSSH (single bit difference in binary)
- Kernel module modifying source code when "viewed" by GCC only (see
media.ccc.de
)
- Compromised Apple iOS SDK, Xcodeghost, etc.
The motivation behind "reproducible" builds is to allow verification
that no flaws have been introduced during the compilation process.
The solution
- Ensure compilation always identical results
- Multiple parties compare compilation results
- Attacker needs to infect everybody simultaneously (or they are detected)
Challenges
- Timestamps
- Timezones & locales
- Non-deterministic file ordering
- Dictionary/hash key ordering
- Users, groups,
umask
, environment variables
- Build paths
- Specifying the environment
Technical advantages
- Faster to build; saves time, money & the environment
- Easier to test changes/revisions
- Unsafe behaviour (eg. internet access)
- Unreliable / non-deterministic behaviours (eg. timing)
- Finds bugs in uncommon timezones or locales
- Detect corrupted build environments
- Find future build failures (eg. expired certificates)
Reproducible builds in Debian
Continuously build every package twice, varying:
- Time & date
- Hostname & domain name
- Filesystem (
disorderfs
)
- Timezone & locale
uid
& gid
- GECOS information, the shell & a bunch of environment variables
- Kernel & CPU type
- and more…
Beyond Debian…
- coreboot, Fedora, LEDE, OpenWRT, NetBSD, FreeBSD, Arch, Qubes, F-Droid, NixOS, Guix, etc.
- Other projects now using "our" testing framework,
SOURCE_DATE_EPOCH
, .buildinfo
file concept
- Reproducible Builds summits (Athens, Berlin)
- Some challenges moving from
debian-
prefixes, mailing lists, etc.
- Generic tools
Future work
- dak (
.buildinfo
file support)
- How to make it meaningful for end-users
- Source code still vulnerable
Todays special thanks:
from Debian and from all folks interested in Reproducible Builds!