path: root/TODO

ToDo for jenkins.debian.net
===========================
:Author:           Holger Levsen
:Authorinitials:   holger
:EMail:            holger@layer-acht.org
:Status:           working, in progress
:lang:             en
:Doctype:          article
:Licence:	   GPLv2

== About jenkins.debian.net

See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian.net"] for a general description of the setup. Below is the current TODO list, which is long and probably incomplete too. The links:https://jenkins.debian.net/userContent/contributing.html[the preferred form of contributions] are patches via pull requests.

== Fix user submitted bugs

* There are  link:https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=jenkins;users=qa.debian.org%40packages.debian.org["bugs filed against the pseudopackage 'qa.debian.org' with usertag 'jenkins'"] in the BTS which would be nice to be fixed soon, as some people actually care.

== General ToDo

* replace amd64 in scripts with $HOSTARCH
** alioth... just manually push it
* *backup*:
** gpg encrypted to some keys
** run on alioth or paradis
** '/var/lib/jenkins/jobs' (the results - the configs are in .git)
** '/var/lib/munin'
** '/var/log'
** '/root/' (contains etckeeper.git)
** '/var/lib/jenkins/reproducible.db' (is backed up manually)
** '/srv/jenkins.debian.net-scm-sync.git' (is backed up manually)
** '/var/lib/jenkins/plugins/*.jpi' (can be derived from jdn-scm-sync.git)
** '/srv/jenkins.debian.net-scm-sync.git'
** '/etc/.git' and '/etc'
* (gz|x)-compress jenkins and reproducible logs and teach apache to uncompress them on-fly (maybe uncompress them only if the client does not support compressed content?) - This is also recorded in the "To be done once jenkins.d.n runs jessie" section...
* put /etc/rc.local in git and extend it to do cleanup of lockfiles etc

=== TODO for testing stretch

Most jobs have been converted, a few are left to do:

* add g-i tests for stretch
* add stretch live-builds
* do lvc for stretch too
* mention stretch in README where appropriate

=== move this setup to jenkins.d.o

The plan is to run a jenkins.d.o host, which is maintained by DSA, but we are maintaining jenkins on it (so we can install any plugins we like etc). then we also setup several jenkins slaves, probably/maybe also maintained by DSA (so we get them into their munin), but on which we can use sudo as we need it. (or maybe not dsa-maintained slaves, so that we can use sudo as we need, for the price of not being in DSAs munin.)

==== next steps for jenkins.d.o migration

* weasel/h01ger: install jenkins.deb
** also create jenkins users in jenkins (KISS)
* h01ger: get slaves: wishlist for starting: 3 slaves, 8 cores, 32gb ram, 150gb hd space if we dont need squid3 on them, 200gb if we do.
** install slaves - how (to automate)?
* install jenkins-job-builder
** needs proper package
* ... (to be planned, see below)
* update DNS to point jenkins.d.o to jerea.d.o
* ... (to be planned, see below)

==== unsorted notes for jenkins.d.o migration

* chroot jobs should use real schroot sessions, and not just use schroot as poor chroot(8) replacement. some links:
** https://anonscm.debian.org/cgit/mirror/dsa-puppet.git/tree/modules/schroot
** https://anonscm.debian.org/cgit/mirror/dsa-puppet.git/tree/modules/porterbox/files/dd-schroot-cmd
** https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/build-wrapper
* sudoers.d/jenkins:
** not suitable for jenkins.d.o, thus we will run all tests on slaves, where DSA doesnt care what we do
* upgrade to jessie, software used which is not in jessie / available as jenkins plugin:
** jenkins.deb
*** DSA prefers if we could use jenkins from jessie-backports
*** 2nd option: own repo, only contains jenkins.deb
*** 3rd option: use upstreams repo
** jenkins-job-builder probably needs to be more properly packaged
*** could be installed locally in jenkins home
** livescreenshot plugin (we use a patched version)
*** jenkins maintaince probably is best done by jenkins users (as opposed to DSA) so that's up to us
* munin monitoring of the slaves
** DSA munin configuration is auto generated by puppet, so the slaves should become .d.o hosts too, to be included
* the existing jenkins.d.o host needs to be renamed to something else (thats "just work" to do but not a major obstacle)

=== To be done once jenkins.d.n runs jessie

* replace with bin/setsid.py workaround with setsid from the util-linux package from jessie
* etc/pbuilderrc: remove /run/shm bindmount once we are running jessie - this is a workaround for #700591
* bin/g-i-installation: use lvcreate without --virtualsize
* check if the sudo workaround in bin/g-i-installation is still needed: 'guestmount -o uid=$(id -u) -o gid=$(id -g)' would be nicer, but it doesnt work: as root, the files seem to belong to jenkins, but as jenkins they cannot be accessed.
* reproducible: let the scheduler use the xz-compressed file instead of the gzip one. This is because of missing lzma module in wheezy's python.
* job-cfg/dpkg.yaml: drop source:jenkins-dpkg-jessie, these jobs dont need a schroot

=== To be done once bugs are fixed

* link:https://bugs.debian.org/767260[#767260] workaround in bin/d-i_build.sh (console-setup doesn't support parallel build)
* link:https://bugs.debian.org/767032[#767032] manual fix in etc/munin/plugins/munin_stats
* link:https://bugs.debian.org/767100[#767100] work in progress in etc/munin/plugins/cpu
* link:https://bugs.debian.org/767018[#767018] work in progress in etc/munin/plugins/iostat_ios
* link:https://bugs.debian.org/774685[#774685] workaround in bin/reproducible_create_meta_pkg_sets.sh

=== jenkins-job-builder related

* use jessie version plus my patches from kali
* change of syntax:
----
properties:
  - priority-sorter:
      priority: 150
----
* this seems to be helpful: http://en.wikipedia.org/wiki/YAML#References (pyyaml which jenkins-job-builder uses supports them)
* cleanup my patches (eg add documentation) and send pull requests on github:
** publisher:logparse
** publisher:htmlpublisher
** svn:scm
** wrappers:live-screenshot

=== livescreenshot plugin

* publish forked livescreenshot plugin and send pull request for my bugfix

== lvc, work in progress, just started

* put this on debian isos too: config/chroot_local-includes/lib/live/config/9999-autotest

* add another (smaller) test: download+run torbrowser daily

* re-read the docs! 
** http://live.debian.net/manual/stable/html/live-manual.en.html#321

* generate feature files from templates? to cope with sub-products?
-> no. detect desktop type and set variables accordingly
-> simpler: pass an environment variable with the type

* get iso
* tables for looping through features: see tails/iuk.git/features/download_target_file/Download_Target_File.feature

* to debug cucumber: --verbose --backtrace --expand

* drop / remove
* can probably go: dhcp.rb firewall_leaks.rb dhcp.feature firewall_leaks.feature
* more occurances of "the computer boots Tails"
* @source (only keep product tests)

* disabled stuff in common_steps.rb
** #if @vm.execute("service tor status").success?

* "I set sudo password" not needed for debianlive nor debian(edu):
** #@screen.wait("TailsGreeterAdminPassword.png", 20)

* $misc_files_dir needed?
* def sort_isos_by_creation_date
  Dir.glob("#{Dir.pwd}/*.iso").sort_by {|f| tails_iso_creation_date(f)}
  -> useless for us, purpose is to automatically select the latest iso if none is given
* search case-in-sensitive for tails+tor+amnesia

* put in update_jdn.sh:
----
    addgroup tcpdump
    dpkg-statoverride --update --add root tcpdump 754 /usr/sbin/tcpdump
    setcap CAP_NET_RAW+eip /usr/sbin/tcpdump
    adduser $USER tcpdump
    adduser $USER libvirt
    adduser $USER libvirt-qemu
----

== Improve existing tests

=== reproducible

* issues in packages or debbindiff:
** find debbindiff problems: 
----
egrep -R -l '(debbindiff had trouble comparing|maybe there is still )' /var/lib/jenkins/userContent/rbuild/
----

* packages build with different dependencies, should be rescheduled:
----
grep '(=.*).,.$' rygel_0.24.2-1.debbindiff.html | uniq -c | sort | grep -c '^\s*1 '
----
** if the value is larger than 0, there are differing dependencies...
** the grepping should probably only done on files younger then $check_frequency

* misc
** move all content into /var/lib/userContent/reproducible/
** more graphs: graph average build duration by day
** meld bin/reproducible_setup_schroot.sh into bin/schroot-create.sh and alias the former to the latter
** "fork" etc/schroot/default into etc/schroot/reproducible
** use one css for all, not two minimally different ones
** repo-comparison: check for binaries without source
** reproducible_create_meta_pkg_sets uses schroot created by dpkg_setup_schroot_jessie job (outside of reproducible job space...)

* build related
** Store the two buildlogs in separated (g|x)zipped files and propose a diff of the two

* notes related
** new page with annoted packages without categorized issues
** new page with notes that doesnt make sense: a.) packages which are reproducible but should not, packages that build but shouldn't, etc.
** notes job shouldn't fail when it hits unknown packages (=removed ones + typos) but rather point this out, probably via irc notify

* pkg sets related:
** for all pkg sets: only display FTBR+FTBFS and reproducible packages with notes by default, provide a way to show them all...
** replace bin/reproducible_installed_on_debian.org with a proper data provider from DSA

* db schema 2.0
** move "untested" field in stats table too? (as in csv output...)
** support for irc-notification of test done (without leaving artifacts)

* missing tests: variation in kernel and date and cpu type
** variations in date are only practical to do in c-i fashion using faketime, thus I suggest to wait for those until we test in the wild...
** kernel / uname output is changed (using linux64) and shall be changed more in future by doing the 2nd build on a different build host running a different kernel..., no need to do fancy stuff with building in qemu or such...
** different cpu type: Opteron_G3  AMD Opteron 23xx (Gen 3 Class Opteron) is the most powerful one that's different to current Opteron_G4
*** that 2nd slave can also run with a different date (or year, so probably 1984)

* enable people to upload test packages, to be built in jenkins:
	<mapreri> h01ger: another wild future request by me: allowing us to upload something and let jenkins test it. rationale: I sent (another) patch for debian-keyring, to fix a timestamp issue in debian control files (due to not_using_dh-builddeb), but there is also a umask issue. I don't want to bother me to setup the very same things jenkins tests locally (I already did too much in this regards, imho), but really people can't tests everything 
	<mapreri> jenkins tests.
	<h01ger> mapreri: please add the feature request to the todo. i'm thinking now that it maybe should just be a jenkins job not integrated into the rp.d.n webui, but... maybe we find a nice way to do it
	<mapreri> h01ger: I'm instead thinking about a repo defining a reproducible-specific suite or something on that line, that integrates well with the current setup. but this is really something wild.
	<h01ger> well, and everybody in debian-keyring from sid can uplood? :)
	<mapreri> that would be wonderful.

=== qa.debian.org*

* udd-versionskew: also provide arch-relative version numbers in output too

=== d-i_manual*

* svn:trunk/manual/po triggers the full build, should trigger language specific builds.
* svn:trunk/manual is all thats needed, not whole svn:trunk

=== d-i_build*

* build packages using jenkins-debian-glue and not my hacks?
* run scripts/digress/ ?
* bubulle wrote: "Another interesting target would be d-i builds *including non uploaded packages* (something like "d-i from git repositories" images). That would in some way require to create a quite specific image, with all udebs (while netboot only has udebs needed before one gets a working network setup).

=== chroot-installation_*

* use schroot for chroot-installation, stop using plain chroot everywhere
* add alternative tests with aptitude and possible apt
* split etc/schroot/default
* inform debian-devel@l.d.o or -qa@?
* warn about transitional packages installed (on non-upgrades only)
* install all the tasks "instead", thats rather easy nowadays as all task packages are called "task*".
** make sure this includes blends

=== g-i-installation_*

Development of these tests has stopped. In future the 'lvc*' tests should replace them.

These small changes are probably still worth doing anyway:

* g-i: replace '--' with '---' as param delimiter. see #776763 / 5df5b95908 in d-e-c
* download .isos once in central place
** /var/lib/jenkins/jobs/g-i-installation_*/workspace/*iso needs 53GB currently, it could be 30 less
* g-i_presentation: use preseeding files on jenkins.d.n and not hands.com
* turn job-cfg/g-i.yaml into .yaml.py

The following ideas should really only be implemented for the new 'lvc*' tests.... (but are kept here for now)

* pick LANG from predefined list at random - if last build was not successful or unstable fall back to English
** these jobs would not need to do an install, just booting them in rescue mode is probably enough
* for edu mainservers running as servers for workstations etc: "d-i partman-auto/choose_recipe select atomic" to be able to use smaller disk images
** same usecase:  -monitor none -nographic -serial stdio  

== Further ideas...

=== Test them all

* build packages from all team repos on alioth with jenkins-debian-glue on team request (eg, via a .txt file in a git.repo) for specific branches (which shall also be automated, eg. to be able to only have squeeze+sid branches build, but not all other branches.)

== Debian Packaging related

This setup should come as a Debian source package...

* /usr/sbin/jenkins.debian.net-setup needs to be written
* what update-j.d.n.sh does, needs to be put elsewhere...
* debian/copyright is incorrect about some licenses:
** the profitbricks+debian+jenkins logos
** the preseeding files
** ./feature/ is gpl3

// vim: set filetype=asciidoc: