jenkins.debian.net or what is Debian doing with all these ressources


Holger Levsen (h01ger)

about me

  • Debian user since 1995
  • Debian contributor since 2001
  • DebConf organizer, founded the DebConf video team in 2005
  • Debian developer since 2007
  • Debian-Edu (Debian for Education), since 2003
  • Debian-QA
  • Freelancer at Profitbricks from 2012-2015

more about Debian QA and me

  • https://piuparts.debian.org/ since 2009
  • https://jenkins.debian.net/ since 2012
  • https://reproducible.debian.net/ since 2014
  • since 2015 funded by the Linux Foundation

jenkins.debian.net contributors:

  • Phil Hands: lvc
  • Helmut Grohne: rebootstrap
  • Samuel Thibault: hurd + accessibility
  • Steven Chamberlain: kfreebsd
  • Tomasz Nitecki: jenkins java support
  • Mattia Rizzolo, Valerie Young and others: reproducible Debian
  • several: reproducible other than Debian
  • 36 contributors to jenkins.debian.net.git in total

Profitbricks ressources used by jenkins.debian.net

  • 2 datacenters (FKB + FRA)
  • 17 machines (16*Debian, 1*FreeBSD, 13*64bit, 4*32bit)
  • 168 cores (148 AMD, 20 Intel)
  • 498/503 GB RAM
  • 2.9/3.1 TB HDD storage
  • 2.9/2 TB SDD storage
  • no static IP addresses
  • 2 DCD users: Mattia Rizzolo and me

Package (collection) installation tests

  • 338 jobs
  • wheezy (98), jessie (147), stretch (153), sid (98)
  • upgrades and new installations tested

g-i-installation tests

  • tests Debian Installer (d-i) in graphical mode ("g-i") and text mode too
  • creates videos and screenshots
  • plain Debian (installations and rescue mode) and Debian Edu
  • kfreebsd and hurd
  • finally almost deprecated today
  • replaced by lvc tests (libvirt-cucumber)

Ressources used by reproducible.debian.net, by architecture

  • FIXME: page is out of place here
  • 13 amd64 systems, sponsored by Profitbricks
  • 4 i386 systems, sponsored by Profitbricks
  • 22 armhf systems, sponsored by vagrant@d.o, Debian & other donations
  • soon: 8 arm64 systems, sponsored by codethink.co.uk

The problem

  • Can inspect the source code of free software for flaws
  • But distributions provide binary/compiled packages

Can we trust this process?

  • To get users, go after the developers
  • Financial incentives to crack developer machines / build infrastructure
  • CVE-2002-0083: Remote root exploit in OpenSSH (single bit difference in binary)
  • Kernel module modifying source code when "viewed" by GCC only (see media.ccc.de)
  • Compromised Apple iOS SDK, Xcodeghost, etc.

The motivation behind "reproducible" builds is to allow verification that no flaws have been introduced during the compilation process.

The solution

  1. Ensure compilation always identical results
  2. Multiple parties compare compilation results
  3. Attacker needs to infect everybody simultaneously (or they are detected)

Challenges

    • Timestamps
    • Timezones & locales
    • Non-deterministic file ordering
    • Dictionary/hash key ordering
    • Users, groups, umask, environment variables
    • Build paths
    • Specifying the environment

Technical advantages

    • Faster to build; saves time, money & the environment
    • Easier to test changes/revisions
    • Unsafe behaviour (eg. internet access)
    • Unreliable / non-deterministic behaviours (eg. timing)
    • Finds bugs in uncommon timezones or locales
    • Detect corrupted build environments
    • Find future build failures (eg. expired certificates)

Reproducible builds in Debian

Continuously build every package twice, varying:

    • Time & date
    • Hostname & domain name
    • Filesystem (disorderfs)
    • Timezone & locale
    • uid & gid
    • GECOS information, the shell & a bunch of environment variables
    • Kernel & CPU type
    • and more…
 
 

2016 summit meeting

  • Three-day workshop in Berlin, Germany
  • Follow-up to Athens 2015 event


reproducible-builds.org/events/berlin2016/

Beyond Debian…

  • coreboot, Fedora, LEDE, OpenWRT, NetBSD, FreeBSD, Arch, Qubes, F-Droid, NixOS, Guix, etc.
  • Other projects now using "our" testing framework, SOURCE_DATE_EPOCH, .buildinfo file concept
  • Reproducible Builds summits (Athens, Berlin)
  • Some challenges moving from debian- prefixes, mailing lists, etc.
  • Generic tools
 

Future work

  • dak (.buildinfo file support)
  • How to make it meaningful for end-users
  • Source code still vulnerable

Usual thanks:




Todays special thanks:


from Debian and from all folks interested in Reproducible Builds!

Questions?



holger@debian.org B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

https://jenkins.debian.net
https://reproducible-builds.org