jenkins.debian.net or what is Debian doing with all these ressources


Holger Levsen (h01ger)

about me

  • Debian user since 1995
  • Debian contributor since 2001
  • DebConf organizer, founded the DebConf video team in 2005
  • Debian developer since 2007
  • Debian-Edu (Debian for Education), since 2003
  • Debian-QA
  • Freelancer at Profitbricks from 2012-2015

more about Debian QA and me

  • https://piuparts.debian.org/ since 2009
  • https://jenkins.debian.net/ since 2012
  • https://reproducible.debian.net/ since 2014
  • since 2015 funded by the Linux Foundation

jenkins.debian.net contributors:

  • Phil Hands: lvc
  • Helmut Grohne: rebootstrap
  • Samuel Thibault: hurd + accessibility
  • Steven Chamberlain: kfreebsd
  • Tomasz Nitecki: jenkins java support
  • Mattia Rizzolo, Valerie Young and others: reproducible Debian
  • several: reproducible other than Debian
  • 36 contributors to jenkins.debian.net.git in total

Profitbricks ressources used by jenkins.debian.net

  • 2 datacenters (FKB + FRA)
  • 17 machines (16*Debian, 1*FreeBSD, 13*64bit, 4*32bit)
  • 168 cores (148 AMD, 20 Intel)
  • 498/503 GB RAM
  • 2.9/3.1 TB HDD storage
  • 2.9/2 TB SDD storage
  • no static IP addresses
  • 2 DCD users: Mattia Rizzolo and me

Package (collection) installation tests

  • 338 jobs
  • wheezy (98), jessie (147), stretch (153), sid (98)
  • upgrades and new installations tested

g-i-installation tests

  • tests Debian Installer (d-i) in graphical mode ("g-i") and text mode too
  • creates videos and screenshots
  • plain Debian (installations and rescue mode) and Debian Edu
  • kfreebsd and hurd
  • finally almost deprecated today
  • replaced by lvc tests (libvirt-cucumber)

reproducible.debian.net

  • created by 379 jobs on jenkins.debian.net

tests.reproducible-builds.org/debian/

  • created by ~350 jobs on jenkins.debian.net

The problem: why do we need reproducible builds?

  • One can inspect the source code of free software for flaws
  • But distributions provide binary/compiled packages

Can we trust this process?

  • To get users, go after the developers
  • Financial incentives to crack developer machines / build infrastructure
  • CVE-2002-0083: Remote root exploit in OpenSSH (single bit difference in binary)
  • Kernel module modifying source code when "viewed" by GCC only (see media.ccc.de)
  • Compromised Apple iOS SDK, Xcodeghost, etc.

The motivation behind "reproducible" builds is to allow verification that no flaws have been introduced during the compilation process.

The solution

    FIXME wordings
  1. Ensure compilation always identical results
  2. Multiple parties compare compilation results
  3. Attacker needs to infect everybody simultaneously (or they are detected)

We call this Reproducible Builds.

  • We think this should become the norm for free software.

Reproducible builds in Debian

Continuously build every package twice, varying:

    • Time & date
    • Hostname & domain name
    • Filesystem (disorderfs)
    • Timezone & locale
    • uid & gid
    • GECOS information, the shell & a bunch of environment variables
    • Kernel & CPU type
    • and more…

Challenges

    • Timestamps
    • Timezones & locales
    • Non-deterministic file ordering
    • Dictionary/hash key ordering
    • Users, groups, umask, environment variables
    • Build paths
    • Specifying the environment
 
 

Technical advantages

    • Faster to build; saves time, money & the environment
    • Easier to test changes/revisions
    • Unsafe behaviour (eg. internet access)
    • Unreliable / non-deterministic behaviours (eg. timing)
    • Finds bugs in uncommon timezones or locales
    • Detect corrupted build environments
    • Find future build failures (eg. expired certificates)

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

https://try.diffoscope.org

Future work

  • .buildinfo files distribution unsolved
  • How to make it meaningful for end-users
  • Source code still vulnerable
 

Beyond Debian…

  • Reproducible Builds summits (Athens 2015, Berlin 2016)

Ressources used by reproducible.debian.net, by architecture & sponsor

  • 13 amd64 systems, sponsored by Profitbricks
  • 4 i386 systems, sponsored by Profitbricks
  • 22 armhf systems, sponsored by vagrant@d.o, Debian & other donations
  • soon: 8 arm64 systems, sponsored by codethink.co.uk

Usual thanks:




Todays special thanks:

  • from Debian, jenkins.debian.net would not have been possible like this without your support!
  • from many many folks interested in Reproducible Builds

Questions?



holger@debian.org B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

https://jenkins.debian.net
https://reproducible-builds.org