jenkins.debian.net or what is Debian doing with all these ressources

(Automate all the tests!)


Holger Levsen (h01ger)

about me

  • Debian user since 1995
  • Debian contributor since 2001
  • DebConf organizer, founded the DebConf video team in 2005
  • Debian developer since 2007, holger@debian.og
  • Debian-Edu (Debian for Education), since 2003
  • Debian-QA
  • Freelancer at Profitbricks from 2012-2015

more about Debian QA and me

  • https://piuparts.debian.org since 2009 - today juggling with 624234 logs from 53158 packages in 28 suites with Andreas Beckmann
  • https://jenkins.debian.net since 2012
  • https://reproducible.debian.net since 2014
  • since 2015 funded by the Linux Foundation for working on https://reproducible-builds.org

about jenkins.debian.net

  • ressources sponsored by Profitbricks since 2012
 

Profitbricks ressources used by jenkins.debian.net

  • 17 machines (16*Debian, 1*FreeBSD, 13*64bit, 4*32bit)
    in 2 datacenters (FKB + FRA)
  • 168 cores (148 AMD, 20 Intel) with 498/503 GB RAM
  • 2.9/3.1 TB HDD and 1.9/2 TB SDD storage
  • no static IP addresses, no idea about traffic (500gb/month?)
  • 2 DCD users: Mattia Rizzolo and me

jenkins.debian.net contributors:

  • Phil Hands: lvc
  • Helmut Grohne: rebootstrap
  • Samuel Thibault: hurd + accessibility
  • Steven Chamberlain: kfreebsd
  • Tomasz Nitecki: jenkins java support
  • Mattia Rizzolo, Valerie Young and others: reproducible Debian
  • 36 contributors to jenkins.debian.net.git in total

A quick detour about Debian release names

  • wheezy (Debian 7) = oldstable
  • jessie (8) = stable
  • stretch (9 = testing
  • sid = unstable
  • experimental

Package (collection) installation tests

  • 338 jobs
  • wheezy (98), jessie (147), stretch (153), sid (98)
  • upgrades and new installations tested

g-i-installation tests

  • tests Debian Installer (d-i) in graphical mode ("g-i") and text mode too
  • creates videos and screenshots
  • plain Debian (installations and rescue mode) and Debian Edu
  • kfreebsd and hurd
  • finally almost deprecated today
  • replaced by lvc tests (libvirt-cucumber)

reproducible.debian.net

  • created by 379 jobs on jenkins.debian.net

tests.reproducible-builds.org/debian/

  • created by ~350 jobs on jenkins.debian.net

The problem: why do we need reproducible builds?

  • One can inspect the source code of free software for flaws
  • But distributions provide binary/compiled packages

Can we trust this process?

  • To get users, go after the developers
  • Financial incentives to crack developer machines / build infrastructure
  • CVE-2002-0083: Remote root exploit in OpenSSH (single bit difference in binary)
  • Kernel module modifying source code when "viewed" by GCC only (see media.ccc.de)
  • Compromised Apple iOS SDK, Xcodeghost, etc.

The motivation behind "reproducible" builds is to allow verification that no flaws have been introduced during the compilation process.

The solution - FIXME wordings

  • Ensure compilation always identical results
  • Multiple parties compare compilation results
  • Attacker needs to infect everybody simultaneously (or they are detected)

We call this Reproducible Builds.

  • We think this should become the norm for free software.

Reproducible builds in Debian

Continuously build every package twice, varying:

    • Time & date
    • Hostname & domain name
    • Filesystem (disorderfs)
    • Timezone & locale
    • uid & gid
    • GECOS information, the shell & a bunch of environment variables
    • Kernel & CPU type
    • and more…

Challenges

    • Timestamps
    • Timezones & locales
    • Non-deterministic file ordering
    • Dictionary/hash key ordering
    • Users, groups, umask, environment variables
    • Build paths
    • Specifying the environment
 
 

Technical advantages

    • Faster to build; saves time, money & the environment
    • Easier to test changes/revisions
    • Unsafe behaviour (eg. internet access)
    • Unreliable / non-deterministic behaviours (eg. timing)
    • Finds bugs in uncommon timezones or locales
    • Detect corrupted build environments
    • Find future build failures (eg. expired certificates)

 
 
 
 
 
 
 
 
 
 
 
 
 

https://try.diffoscope.org

Future work

  • .buildinfo files distribution unsolved
  • How to make it meaningful for end-users
  • Source code still vulnerable
 

Beyond Debian…

Reproducible Builds summits (Athens 2015, Berlin 2016)

Projects using Profitbricks ressources via jenkins.debian.net

works:

worked:

work in progress:

Ressources used by reproducible.debian.net, by architecture & sponsor

  • 13 amd64 systems, sponsored by Profitbricks
  • 4 i386 systems, sponsored by Profitbricks
  • 22 armhf systems, sponsored by vagrant@d.o, Debian & other donations
  • soon: 8 arm64 systems, sponsored by codethink.co.uk

Usually I thank:




Todays special thanks:

  • from Debian, jenkins.debian.net would not have been possible like this without your support!
  • from many many folks interested in Reproducible Builds!

Questions?



holger@debian.org B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

https://jenkins.debian.net
https://reproducible-builds.org





…and many thanks to Chris Lamb for the nice slides design!