From fa33733fc6e8050631e6e09202ca92047c98697d Mon Sep 17 00:00:00 2001 From: Holger Levsen Date: Wed, 30 Nov 2016 21:24:36 +0100 Subject: presenation: move to userContent/ so it becomes visible Signed-off-by: Holger Levsen --- .../2016-11-30-Profitbricks/index.html | 557 +++++++++++++++++++++ 1 file changed, 557 insertions(+) create mode 100644 userContent/presentations/2016-11-30-Profitbricks/index.html (limited to 'userContent/presentations/2016-11-30-Profitbricks/index.html') diff --git a/userContent/presentations/2016-11-30-Profitbricks/index.html b/userContent/presentations/2016-11-30-Profitbricks/index.html new file mode 100644 index 00000000..b8c1ddea --- /dev/null +++ b/userContent/presentations/2016-11-30-Profitbricks/index.html @@ -0,0 +1,557 @@ + + + + + + jenkins.debian.net or what is Debian doing with all these resources + + + + + + + + + + + + + + + + + + + +
+
+
+

+ about jenkins.debian.net - or what Holger / Debian is doing with all these resources +

+

(Automating all the tests!)

+

+
+ Holger Levsen <holger@debian.org> +

+

Profitbricks Office, 2016-11-30, Berlin, Germany

+
+ +
+

about me

+
    +
  • Debian user since 1995, contributing since 2001
    • +
  • Debian-Edu (Debian for Education), since 2003
    • +
  • DebConf organizer, founded the DebConf video team in 2005
    • +
  • Debian developer since 2007, holger@debian.og
    • +
  • Freelancer since 2004, holgerlevsen.de
    • +
  • Freelancer at Profitbricks from 2011-2013 and 2015
    • +
+
+ +
+

more about Debian QA and me

+
    +
  • https://piuparts.debian.org since 2009 - today juggling with 648988 logs from 53158 packages in 28 suites with Andreas Beckmann
    • +
  • https://jenkins.debian.net since 2012
    • +
  • https://reproducible.debian.net since 2014
    • +
  • since 2015 funded by the Linux Foundation for working on https://reproducible-builds.org
    • +
+
+ +
+

other Debian QA efforts elsewhere

+
    +
  • lintian.debian.org
    • +
  • ci.debian.net
    • +
  • periodic full archive rebuilds (amd64 only)
    • +
+
+ + +
+

+ about jenkins.debian.net +

+
    +
  • resources sponsored by Profitbricks since 2012
    • +
  • first request on August 2nd 2012: 2-4 cores, 2GB RAM, 1 TB storage
    • +
+
+ +
+   +
+ +
+

+ Profitbricks resources used by jenkins.debian.net +

+
    +
  • 17 machines (16*Debian, 1*FreeBSD, 13*64bit, 4*32bit)
    + in 2 datacenters (FKB + FRA)
    • +
  • 168 cores (148 AMD, 20 Intel) with 498/503 GB RAM
    • +
  • 2.9/3.1 TB HDD and 1.9/2 TB SDD storage
    • +
  • no static IP addresses, no idea about traffic (500gb/month?)
    • +
  • 2 DCD users: Mattia Rizzolo and me
    • +
+
+ + +
+

+ jenkins.debian.net contributors: +

+
    +
  • Mattia Rizzolo, Valerie Young and others: reproducible Debian
    • +
  • Helmut Grohne: rebootstrap
    • +
  • Samuel Thibault: hurd + accessibility
    • +
  • Steven Chamberlain: kfreebsd
    • +
  • Phil Hands: lvc
    • +
  • Tomasz Nitecki: jenkins java support
    • +
  • 36 contributors to jenkins.debian.net.git in total
    • +
+
+ + +
+

+ A quick detour about Debian release names +

+
    +
  • wheezy (Debian 7) = oldstable
    • +
  • jessie (8) = stable
    • +
  • stretch (9 = testing
    • +
  • sid = unstable
    • +
  • experimental
    • +
+
+ +
+

+ Benefits for debian +

+
    +
  • + IRC / mail notifications, #debian-qa IRC channel and 11 other channels, more mailinglists +
    • +
  • + Early notifications of problems - though bugs still need to be filed manually +
    • +
  • + Countless bug reports, ie 2670 done ftbfs via reproducible… (and 500 open…) +
    • +
  • + jenkins.debian.net will become an offical service, jenkins.debian.org +
    • +
+
+ +
+

+ chroot-installation tests +

+
    +
  • 338 jobs basically running apt install $metapackages
    + (gnome, kde, cinnamon, lxde, xfce, qt4, qt5, haskell, developer, debconf-video, debian-edu)
    • +
  • new installations and upgrades tested in
    + wheezy (98), jessie (147), stretch (153), sid (98)
    • +
  • wheezy monthly, jessie weekly, stretch every other day, sid daily
    • +
+
+ +
+

+ g-i-installation tests +

+
    +
  • tests Debian Installer (d-i) in graphical mode ("g-i") and text mode
    • +
  • creates videos and screenshots
    • +
  • plain Debian (installations and rescue mode) and Debian Edu
    • +
  • jessie, stretch and sid
    • +
  • linux, kfreebsd and hurd
    • +
  • finally almost deprecated today, will be replaced by lvc tests (libvirt-cucumber) maintained by Phil Hands
    • +
+
+ +
+

+ more debian-installer related jobs: +

+
    +
  • 97 packages (building udebs) triggered by commits to their git master branches
    • +
  • manual in 24 languages, also git triggered
    • +
  • lvc and d-i from proposed branches planned
    • +
+
+ +
+

+ 37 debian-edu jobs: +

+
    +
  • + 28 g-i tests for jessie and stretch +
    • +
  • + 8 debian-edu packages build triggerd by commits on their git master branches +
    • +
  • + very useful for debian-edu-doc which is published for 7 languages in HTML, PDF & EPUB format. +
    • +
+
+ + +
+

+ rebootstrap: +

+
    +
  • Cross bootstrap Debian from scratch…
    • +
  • alpha arm64 arm64ilp32 armel armhf hppa hurd-amd64 hurd-i386 i386 kfreebsd-amd64 kfreebsd-armhf kfreebsd-i386 m68k mips mips64el mips64r6el mipsel mipsr6el musl-linux-arm64 musl-linux-armhf musl-linux-i386 musl-linux-mips musl-linux-mipsel nios2 powerpc powerpcel powerpcspe ppc64 ppc64el s390x sh4 sparc sparc64 tilegx x32
    • +
  • Helmut Grohne files lots of cross-building and bootstrapping bugs.
    • +
+
+ +
+

+ more debian-qa related jobs: +

+
  • orphaned packages without bug
    • +
  • dpkg trigger cycles
    • +
  • lintian, debhelper and piuparts are build on git commits in jessie, stretch and unstable
    • +
  • + reproducible-builds.org - "btw": over 2600 'FTBFS' bugs found and fixed so far, ~400 open… +
    • + +
    + + +
    +

    + reproducible.debian.net / tests.reproducible-builds.org/debian/ +

    +
      +
    • created by 379 / ~350 jobs on jenkins.debian.net
      • +
    • it's not only about Debian anymore…
      • +
    +
    + + +
    +

    The problem: Can we trust the build process?

    +
      +
    • One can inspect the source code of free software for flaws
      • +
    • But distributions provide binary/compiled packages
      • +
    +
    + +
    +

    The problem: nobody can trust any binary built anywhere anymore

    + +
      +
    • To get users, go after the developers
      • +
    • Financial incentives to crack developer machines / build infrastructure
      • +
    • CVE-2002-0083: Remote root exploit in OpenSSH (single bit difference in binary)
      • +
    • Kernel module modifying source code when "viewed" by GCC only (see media.ccc.de)
      • +
    • Compromised Apple iOS SDK, Xcodeghost, etc.
      • +
    +
    + +
    +

    Our solution

    +
      (we are still at step 1 here) +
    • Ensure compilation of the same source always has bit by bit identical results
      • +
    • Multiple parties compare compilation results
      • +
    • Attacker needs to infect everybody simultaneously (or they are detected)
      • +
    +
    + +
    +

    We call this Reproducible Builds.

    + +
      +
    • We think this should become the norm for free software.
      • +
    +
    + +
    +

    + The motivation behind "reproducible" builds is to allow verification + that no flaws have been introduced during the compilation process. +

    +
    + + +
    +

    Reproducible builds in Debian

    + +

    Continuously build every package twice, varying:

    + +
      +
        +
      • Time & date
        • +
      • Hostname & domain name
        • +
      • Filesystem (disorderfs)
        • +
      • Timezone & locale
        • +
      • uid & gid
        • +
      • GECOS information, the shell & a bunch of environment variables
        • +
      • Kernel & CPU type
        • +
      • and more…
        • +
      +
    +
    + +
    +

    +  
    +  
    +  
    +  
    +  
    +  
    +  
    +  
    +  
    +  
    +  
    +  
    +  
    +

    https://try.diffoscope.org

    +

    +
    + +
    +

    Challenges

    +
      +
        +
      • Timestamps
        • +
      • Timezones & locales
        • +
      • Non-deterministic file ordering
        • +
      • Dictionary/hash key ordering
        • +
      • Users, groups, umask, environment variables
        • +
      • Build paths
        • +
      • Specifying the environment
        • +
      +
    +
    + +
    +   +
    + +
    +   +
    + +
    +

    Other technical benefits

    + +
      +
        +
      • Faster to build; saves time, money & the environment
        • +
      • Easier to test changes/revisions
        • +
      • Unsafe behaviour (eg. internet access)
        • +
      • Unreliable / non-deterministic behaviours (eg. timing)
        • +
      • Finds bugs in uncommon timezones or locales
        • +
      • Detect corrupted build environments
        • +
      • Find future build failures (eg. expired certificates)
        • +
      +
    +
    + + +
    +   +
    + + +
    +

    Future work

    + +
      +
    • .buildinfo files distribution unsolved (step 2)
      • +
    • How to make it meaningful for end-users (step 3)
      • +
    • Source code still vulnerable
      • +
    +
    + +
    +

    Beyond Debian…

    +

    + + + + + + + + + + + + + + + + + + + +

    Reproducible Builds summits (Athens 2015, Berlin 2016) +
    +

    +
    + +
    +

    Projects using Profitbricks resources via jenkins.debian.net

    +

    works: + + + + + + +

    +

    worked: + + +

    +

    work in progress: + + +

    +
    + +
    +

    + Resources used for reproducibility testing on jenkins.debian.net, by architecture & sponsor +

    +
      +
    • 13 amd64 systems, sponsored by Profitbricks
      • +
    • 4 i386 systems, sponsored by Profitbricks
      • +
    • 22 armhf systems, sponsored by vagrant@d.o, Debian & other donations
      • +
    • soon: 8 arm64 systems, sponsored by codethink.co.uk
      • +
    +
    + + +
    +

    Usually I thank:

    +

    + +
    + +
    + +
    + +

    +
    + +
    +

    Todays special thanks:

    +

    + +

      +
    • from Debian, jenkins.debian.net would not have been possible like this without your support!
      • +
    • from many many folks interested in Reproducible Builds!
      • +
    +

    +
    + +
    +

    Questions?

    + +
    +
    + + + + + + +
    + holger@debian.org + + B8BF 5413 7B09 D35C F026
    FE9D 091A B856 069A AA1C     +
    + +
    + +

    + https://jenkins.debian.net +
    + https://reproducible-builds.org +
    +
    +
    +
    +
    +
    +

    +

    + …and many thanks to Chris Lamb for the nice slide design! +

    +
    +
    +
    + + + + + + -- cgit v1.2.3-70-g09d2