let's maintain jenkins.debian.org as a team


Holger Levsen <holger@debian.org>

DebConf17, 2017-08-07, Montreal, Canada

about me

  • Debian user since 1995, contributing since 2001
  • Debian-Edu (Debian for Education), since 2003
  • DebConf organizer, founded the DebConf video team in 2005
  • Debian developer since 2007, holger@debian.og
  • Freelancer since 2004

more about Debian QA and me

  • https://piuparts.debian.org since 2009 - today (Nov 2016) juggling with 648988 logs from 53158 packages in 28 suites with Andreas Beckmann
  • https://jenkins.debian.net since 2012
  • https://reproducible.debian.net since 2014
  • since 2015 funded by the Linux Foundation for working on https://reproducible-builds.org

I'm busy.

  • so I merge patches very easily, just looking whether it's safe to apply and at most only breaks your stuff.
  • 1565 different jobs with contributions from 40 people, it's a mess. please send patches! either to cleaning up some parts or to add the next little island…
  • "I don't care."

I do care.

  • I really want to maintain this as a team and there is a team already. But it's still too much my setup…
  • There is jenkins.debian.org.
  • Let's go there. This is my outline, please hack it and send patches!

jenkins.debian.net contributor #1

  • Mattia Rizzolo, the only other person having an overview about the whole thing. Has access to everything I have.

jenkins.debian.net contributors

  • Valerie Young and others: reproducible Debian
  • Helmut Grohne: rebootstrap
  • Samuel Thibault: hurd + accessibility
  • Steven Chamberlain: kfreebsd
  • Phil Hands: lvc
  • Tomasz Nitecki: jenkins java support
  • 36 contributors to jenkins.debian.net.git in total, also committers from Arch Linux, openSUSE, LEDE, coreboot, Guix, FreeBSD and NetBSD

Who are you?

  • Who uses jenkins.d.n regularily?
  • Who wants to contribute/control (more) jobs on jenkins.d.o?
  • Who thinks current jenkins is too noisy?

What is jenkins?

  • to me: mostly a cron scheduler with a web UI and notifications.
  • (in)security is part of the design. I only trust jenkins for QA and presenting results, not for generating binaries…

other Debian QA efforts elsewhere

  • lintian.debian.org
  • piuparts.debian.org
  • ci.debian.net
  • periodic full archive rebuilds (amd64 only)
  • ...

about jenkins.debian.net

  • resources sponsored by Profitbricks since 2012
  • first request on August 2nd 2012: 2-4 cores, 2GB RAM, 1 TB storage
 

Profitbricks resources used by jenkins.debian.net

  • 17 machines (16*Debian, 1*FreeBSD, 13*64bit, 4*32bit)
    in 2 datacenters (FKB + FRA)
  • 168 cores (144 AMD, 24 Intel) with 498 GB RAM
  • 3 TB HDD and 2 TB SDD storage
  • no static IP addresses, no idea about traffic (500gb/month?)
  • 2 DCD users: Mattia Rizzolo and me

Resources used for reproducibility testing on jenkins.debian.net, by architecture & sponsor

  • 13 amd64 systems, sponsored by Profitbricks
  • 4 i386 systems, sponsored by Profitbricks
  • 26 armhf systems, sponsored by vagrant@d.o, Debian & other donations
  • 8 arm64 systems, sponsored by codethink.co.uk

about jenkins.debian.net continued…

  • git clone ssh://git.debian.org/git/qa/jenkins.debian.net.git
  • INSTALL, README, job-cfg/*, bin/*
  • code base grown in 5 years by 40 people…
  • ansible, builds with sbuild or jenkins-debian-glue, namespaces, you name it, please send patches
  • views so far were configured manually, but it's now possible with jenkins-job-builder too - please send patches!

Benefits for debian

  • IRC / mail / web notifications, #debian-qa IRC channel and 11 other channels, more mailinglists
  • Early notifications of problems - though bugs still need to be filed manually
  • Countless bug reports, ie 2670 done ftbfs via reproducible…

Benefits for you?

  • The following is a quick summary of the >1000 jobs jenkins is running. Nobody except Mattia and me are expected to remember them all…
  • Please yell if there is a job (notification) annoying you.

misc debian-qa related jobs:

  • orphaned packages without bug
  • dpkg trigger cycles
  • debhelper, debsums, lintian and piuparts are build on git commits in jessie, stretch and unstable
  • multiarch versionskew
  • pkg-haskell

    • chroot-installation tests

    • 453 jobs basically running apt install $metapackages
      (gnome, kde, cinnamon, lxde, xfce, qt4, qt5, haskell, developer, debconf-video, debian-edu, debian-parl, debian-design)
    • new installations and upgrades tested in
      jessie (85), stretch (226), buster (182), sid (184)
    • jessie monthly, stretch weekly, buster every other day, sid daily

    g-i-installation tests

    • tests Debian Installer (d-i) in graphical mode ("g-i") and text mode
    • creates videos and screenshots
    • plain Debian (installations and rescue mode) and Debian Edu
    • jessie, stretch and sid
    • linux, kfreebsd and hurd
    • finally almost deprecated today, will be replaced by lvc tests (libvirt-cucumber) maintained by Phil Hands
     

    lvc = libvirt cucumber

    • lvc tests are work in progress by Phil Hands
    • taken from Tails
    • now I've heard Tails wants to push us using those to improve the quality of Debian to improve the quality of Tails
    • not there yet, but they've started to get useful
    • should fully replace g-i-installation.sh at the end of the year?

    more debian-installer related jobs:

    • 97 packages (building udebs) triggered by commits to their git master branches
    • manual in 24 languages, also git triggered
    • lvc and d-i from proposed branches planned

    37 debian-edu jobs:

    • 28 g-i tests for jessie and stretch
    • 8 debian-edu packages build triggerd by commits on their git master branches
    • very useful for debian-edu-doc which is published for 7 languages in HTML, PDF & EPUB format.

    rebootstrap:

    • Cross bootstrap Debian from scratch…
    • alpha arm64 arm64ilp32 armel armhf hppa hurd-amd64 hurd-i386 i386 kfreebsd-amd64 kfreebsd-armhf kfreebsd-i386 m68k mips mips64el mips64r6el mipsel mipsr6el musl-linux-arm64 musl-linux-armhf musl-linux-i386 musl-linux-mips musl-linux-mipsel nios2 powerpc powerpcel powerpcspe ppc64 ppc64el s390x sh4 sparc sparc64 tilegx x32
    • Helmut Grohne files lots of cross-building and bootstrapping bugs.

    reproducible.debian.net / tests.reproducible-builds.org/debian/

    • created by 483 jobs on jenkins.debian.net
    • we had 151 more jobs but those now run via a systemd service written in shell…/
    • it's not only about Debian anymore…
    • though let's rather talk about jenkins.debian.org now…

    So, about the to migration jenkins.debian.org

    • jerea.debian.org is ready and jenkins.debian.org is a CNAME pointing to it - thanks to DSA!
    • next: create another small script, to deploy (and only deploy) job-cfg/self.yaml
    • then: move on to the next yaml file, eg. piuparts.yaml or whatever. Then: rinse, repeat, until all are moved
    • jenkins.debian.net will become profitbricks-build0 and keep running the actual jobs (and the reproducible webpages) for now…

    Ways to contribute

    • please file bugs about all the issues jenkins finds
    • please file bugs about all the noise jenkins is doing
    • please send patches
    • we have ressources "for anything": write a useful test - and maintain it - and we will happily run it - and don't worry, noone has a test setup except Phil ;-)

    Contact us

    • IRC: #debian-qa on irc.oftc.net
    • Bugs: bugs.debian.org/jenkins.debian.org
    • Lists: jenkins-dev@lists.alioth.debian.org

    Thanks:

       


    Questions?



    holger@debian.org B8BF 5413 7B09 D35C F026
    FE9D 091A B856 069A AA1C

    https://jenkins.debian.net
    git.debian.org/git/qa/jenkins.debian.net.git





    …and many thanks to Chris Lamb for the nice slide design!