about jenkins.debian.net - or what is Holger / Debian doing with all these ressources

(Automating all the tests!)

Holger Levsen <holger@debian.org>

about me

• Debian user since 1995, contributing since 2001
• Debian-Edu (Debian for Education), since 2003
• DebConf organizer, founded the DebConf video team in 2005
• Debian developer since 2007, holger@debian.og
• Freelancer since 2004, holgerlevsen.de
• Freelancer at Profitbricks from 2011-2013 and 2015

more about Debian QA and me

• https://piuparts.debian.org since 2009 - today juggling with 624234 logs from 53158 packages in 28 suites with Andreas Beckmann
• https://jenkins.debian.net since 2012
• https://reproducible.debian.net since 2014
• since 2015 funded by the Linux Foundation for working on https://reproducible-builds.org

about jenkins.debian.net

• ressources sponsored by Profitbricks since 2012

Profitbricks ressources used by jenkins.debian.net

• 17 machines (16*Debian, 1*FreeBSD, 13*64bit, 4*32bit)
  in 2 datacenters (FKB + FRA)
• 168 cores (148 AMD, 20 Intel) with 498/503 GB RAM
• 2.9/3.1 TB HDD and 1.9/2 TB SDD storage
• no static IP addresses, no idea about traffic (500gb/month?)
• 2 DCD users: Mattia Rizzolo and me

jenkins.debian.net contributors:

• Mattia Rizzolo, Valerie Young and others: reproducible Debian
• Helmut Grohne: rebootstrap
• Samuel Thibault: hurd + accessibility
• Steven Chamberlain: kfreebsd
• Phil Hands: lvc
• Tomasz Nitecki: jenkins java support
• 36 contributors to jenkins.debian.net.git in total

A quick detour about Debian release names

• wheezy (Debian 7) = oldstable
• jessie (8) = stable
• stretch (9 = testing
• sid = unstable
• experimental

chroot-installation tests

• 338 jobs basically running apt install $metapackages
  (gnome, kde, cinnamon, lxde, xfce, qt4, qt5, haskell, developer, debconf-video, debian-edu)
• new installations and upgrades tested in
  wheezy (98), jessie (147), stretch (153), sid (98)

g-i-installation tests

• tests Debian Installer (d-i) in graphical mode ("g-i") and text mode
• creates videos and screenshots
• plain Debian (installations and rescue mode) and Debian Edu
• jessie, stretch and sid
• linux, kfreebsd and hurd
• finally almost deprecated today, will be replaced by lvc tests (libvirt-cucumber)

more debian-installer related jobs on jenkins.debian.net:

• FIXME 78 packages (building udebs) triggered by commits to their git master branches
• manual in several/FIXME languages, also git triggered
• lvc and d-i from proposed branches planned

$amount/FIXME debian-edu jobs on jenkins.debian.net:

• g-i tests for jessie and stretch
• six debian-edu packages build triggerd by commits on their git master branches
• very useful for debian-edu-doc which is published for 7 languages in HTML, PDF & EPUB format.

more QA:

• debian-qa
  • orphaned packages without bug
  • dpkg trigger cycles
  • lintian, debhelper and piuparts are build on git commits in jessie, stretch and unstable
• rebootstrap - Helmut Grohne files lots of cross-building and bootstrapping bugs
• reproducible-builds.org - "btw": over 2600 'FTBFS' bugs found and fixed so far, ~400 open…

reproducible.debian.net / tests.reproducible-builds.org/debian/

• created by 379 / ~350 jobs on jenkins.debian.net
• it's not only about Debian anymore…

The problem: Can we trust the build process?

• One can inspect the source code of free software for flaws
• But distributions provide binary/compiled packages

The problem: nobody can trust any binary built anywhere anymore

• To get users, go after the developers
• Financial incentives to crack developer machines / build infrastructure
• CVE-2002-0083: Remote root exploit in OpenSSH (single bit difference in binary)
• Kernel module modifying source code when "viewed" by GCC only (see media.ccc.de)
• Compromised Apple iOS SDK, Xcodeghost, etc.

The motivation behind "reproducible" builds is to allow verification that no flaws have been introduced during the compilation process.

The solution - FIXME wordings

• Ensure compilation always bit by bit identical results
• Multiple parties compare compilation results
• Attacker needs to infect everybody simultaneously (or they are detected)

We call this Reproducible Builds.

• We think this should become the norm for free software.

Reproducible builds in Debian

Continuously build every package twice, varying:

• Time & date
• Hostname & domain name
• Filesystem (disorderfs)
• Timezone & locale
• uid & gid
• GECOS information, the shell & a bunch of environment variables
• Kernel & CPU type
• and more…

Challenges

• Timestamps
• Timezones & locales
• Non-deterministic file ordering
• Dictionary/hash key ordering
• Users, groups, umask, environment variables
• Build paths
• Specifying the environment

Technical benefits

• Faster to build; saves time, money & the environment
• Easier to test changes/revisions
• Unsafe behaviour (eg. internet access)
• Unreliable / non-deterministic behaviours (eg. timing)
• Finds bugs in uncommon timezones or locales
• Detect corrupted build environments
• Find future build failures (eg. expired certificates)

https://try.diffoscope.org

Future work

• .buildinfo files distribution unsolved
• How to make it meaningful for end-users
• Source code still vulnerable

Beyond Debian…

Projects using Profitbricks ressources via jenkins.debian.net

works:

worked:

work in progress:

Ressources used by reproducible.debian.net, by architecture & sponsor

• 13 amd64 systems, sponsored by Profitbricks
• 4 i386 systems, sponsored by Profitbricks
• 22 armhf systems, sponsored by vagrant@d.o, Debian & other donations
• soon: 8 arm64 systems, sponsored by codethink.co.uk

Usually I thank:

Todays special thanks:

• from Debian, jenkins.debian.net would not have been possible like this without your support!
• from many many folks interested in Reproducible Builds!

Questions?

https://jenkins.debian.net
https://reproducible-builds.org

…and many thanks to Chris Lamb for the nice slides design!