ToDo for jenkins.debian.net =========================== :Author: Holger Levsen :Authorinitials: holger :EMail: holger@layer-acht.org :Status: working, in progress :lang: en :Doctype: article :Licence: GPLv2 == About jenkins.debian.net See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian.net"] for a general description of the setup. Below is the current TODO list, which is long and probably incomplete too. The links:https://jenkins.debian.net/userContent/contributing.html[the preferred form of contributions] are patches via pull requests. == Fix user submitted bugs * There are link:https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=jenkins;users=qa.debian.org%40packages.debian.org["bugs filed against the pseudopackage 'qa.debian.org' with usertag 'jenkins'"] in the BTS which would be nice to be fixed soon, as some people actually care. == General ToDo * replace amd64 in scripts with $HOSTARCH * put /etc/rc.local in git and extend it to do cleanup of lockfiles etc * explain in README how to write jobs, eg which pathes are on tmpfs === proper backup * postpone til we run on .debian.org? * gpg encrypted to some keys * run on alioth or paradis * '/var/lib/jenkins/jobs' (the results - the configs are in .git) * '/var/lib/munin' * '/var/log' * '/root/' (contains etckeeper.git) * '/var/lib/jenkins/reproducible.db' (is backed up manually) * '/srv/jenkins.debian.net-scm-sync.git' (is backed up manually) * '/var/lib/jenkins/plugins/*.jpi' (can be derived from jdn-scm-sync.git) * '/srv/jenkins.debian.net-scm-sync.git' * '/etc/.git' and '/etc' === TODO for testing stretch Most jobs have been converted, a few are left to do: * add g-i tests for stretch * add stretch live-builds * do lvc for stretch too * mention stretch in README where appropriate === move this setup to jenkins.d.o The plan is to run a jenkins.d.o host, which is maintained by DSA, but we are maintaining jenkins on it (so we can install any plugins we like etc). then we also setup several jenkins slaves, probably/maybe also maintained by DSA (so we get them into their munin), but on which we can use sudo as we need it. (or maybe not dsa-maintained slaves, so that we can use sudo as we need, for the price of not being in DSAs munin.) ==== next steps for jenkins.d.o migration * weasel/h01ger: install jenkins.deb ** also create jenkins users in jenkins (KISS) * h01ger: get slaves: wishlist for starting: 3 slaves, 8 cores, 32gb ram, 150gb hd space if we dont need squid3 on them, 200gb if we do. ** install slaves - how (to automate)? * install jenkins-job-builder ** needs proper package * ... (to be planned, see below) * update DNS to point jenkins.d.o to jerea.d.o * ... (to be planned, see below) ==== unsorted notes for jenkins.d.o migration * chroot jobs should use real schroot sessions, and not just use schroot as poor chroot(8) replacement. some links: ** https://anonscm.debian.org/cgit/mirror/dsa-puppet.git/tree/modules/schroot ** https://anonscm.debian.org/cgit/mirror/dsa-puppet.git/tree/modules/porterbox/files/dd-schroot-cmd ** https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/build-wrapper * sudoers.d/jenkins: ** not suitable for jenkins.d.o, thus we will run all tests on slaves, where DSA doesnt care what we do * upgrade to jessie, software used which is not in jessie / available as jenkins plugin: ** jenkins.deb *** DSA prefers if we could use jenkins from jessie-backports *** 2nd option: own repo, only contains jenkins.deb *** 3rd option: use upstreams repo ** jenkins-job-builder probably needs to be more properly packaged *** could be installed locally in jenkins home ** livescreenshot plugin (we use a patched version) *** jenkins maintaince probably is best done by jenkins users (as opposed to DSA) so that's up to us * munin monitoring of the slaves ** DSA munin configuration is auto generated by puppet, so the slaves should become .d.o hosts too, to be included * the existing jenkins.d.o host needs to be renamed to something else (thats "just work" to do but not a major obstacle) === To be done once jenkins.d.n runs jessie * Apply the patch from https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=26;filename=pbuilder.patch;att=1;bug=677666 to the installed pbuilder package(s) ** also the other pbuilder patches * replace with bin/setsid.py workaround with setsid from the util-linux package from jessie * bin/g-i-installation: use lvcreate without --virtualsize * check if the sudo workaround in bin/g-i-installation is still needed: 'guestmount -o uid=$(id -u) -o gid=$(id -g)' would be nicer, but it doesnt work: as root, the files seem to belong to jenkins, but as jenkins they cannot be accessed. * install pbuilder from jessie-backports * install botch from jessie-backports (and remove botch from the reproducible-unstable schroot) === To be done once bugs are fixed * link:https://bugs.debian.org/767260[#767260] workaround in bin/d-i_build.sh (console-setup doesn't support parallel build) * link:https://bugs.debian.org/767032[#767032] manual fix in etc/munin/plugins/munin_stats * link:https://bugs.debian.org/767100[#767100] work in progress in etc/munin/plugins/cpu * link:https://bugs.debian.org/767018[#767018] work in progress in etc/munin/plugins/iostat_ios * link:https://bugs.debian.org/774685[#774685] workaround in bin/reproducible_create_meta_pkg_sets.sh === jenkins-job-builder related * use jessie version plus h01ger's patches from kali * change of syntax: ---- properties: - priority-sorter: priority: 150 ---- * this seems to be helpful: http://en.wikipedia.org/wiki/YAML#References (pyyaml which jenkins-job-builder uses supports them) * cleanup h01ger's patches (eg add documentation) and send pull requests on github: ** publisher:logparse ** publisher:htmlpublisher ** svn:scm ** wrappers:live-screenshot === livescreenshot plugin * publish forked livescreenshot plugin and send pull request for h01ger's bugfix == lvc, work in progress, just started * put this on debian isos too: config/chroot_local-includes/lib/live/config/9999-autotest * add another (smaller) test: download+run torbrowser daily * re-read the docs! ** http://live.debian.net/manual/stable/html/live-manual.en.html#321 * generate feature files from templates? to cope with sub-products? -> no. detect desktop type and set variables accordingly -> simpler: pass an environment variable with the type * get iso * tables for looping through features: see tails/iuk.git/features/download_target_file/Download_Target_File.feature * to debug cucumber: --verbose --backtrace --expand * drop / remove * can probably go: dhcp.rb firewall_leaks.rb dhcp.feature firewall_leaks.feature * more occurances of "the computer boots Tails" * @source (only keep product tests) * disabled stuff in common_steps.rb ** #if @vm.execute("service tor status").success? * "I set sudo password" not needed for debianlive nor debian(edu): ** #@screen.wait("TailsGreeterAdminPassword.png", 20) * $misc_files_dir needed? * def sort_isos_by_creation_date Dir.glob("#{Dir.pwd}/*.iso").sort_by {|f| tails_iso_creation_date(f)} -> useless for us, purpose is to automatically select the latest iso if none is given * search case-in-sensitive for tails+tor+amnesia * put in update_jdn.sh: ---- addgroup tcpdump dpkg-statoverride --update --add root tcpdump 754 /usr/sbin/tcpdump setcap CAP_NET_RAW+eip /usr/sbin/tcpdump adduser $USER tcpdump adduser $USER libvirt adduser $USER libvirt-qemu ---- == Improve existing tests === reproducible * r_build.sh ** keep+display both html+txt output from debbindiff ** Store the two buildlogs in separated gzipped files and propose a diff of the two * issues with debbindiff: ** find debbindiff problems - is this caught by bin/reproducible_breakages.py already? ---- egrep -R -l '(debbindiff had trouble comparing|maybe there is still )' /var/lib/jenkins/userContent/rbuild/ ---- * misc ** html_graphs.sh rewrite: split out html_pkg_sets.py first, thats the easiest and the biggest speed gain ** more graphs: graph average build duration by day ** reproducible_create_meta_pkg_sets uses schroot created by dpkg_setup_schroot_jessie job (outside of reproducible job space...) ** "fork" etc/schroot/default into etc/schroot/reproducible ** repo-comparison: check for binaries without source ** move "untested" field in stats table too? (as in csv output...) ** new page: packages which are orphaned but have a reproducible usertagged patch ** a reproducible_log_grep_by_sql.(py|sh) would be nice, to only grep in packages with a certain status (build in the last X days) ** replace submit form by one without javascript (maybe with more url rewriting) * notes related ** #786396: classify issue by "toolchain" or "package" fix needed ** new page with annoted packages without categorized issues ** new page with notes that doesnt make sense: a.) packages which are reproducible but should not, packages that build but shouldn't, etc. *** aint that covered by reproducible_breakages.py already? no. * pkg sets related: ** fix essential set: currently it only has the ones explicitly marked Essential:yes; they and their dependencies make up the full "essential closure set" (sometimes also called pseudo-essential) ** replace bin/reproducible_installed_on_debian.org with a proper data provider from DSA, eg https://anonscm.debian.org/cgit/mirror/debian.org.git/plain/debian/control ** add lua package set (all packages build-depending on dh-lua and starting with lua*) * new script to schedule set of packages: *** all/unreproducible packages affected by a particular issue ** add --debug for the manual scheduler, so that we get notified on irc when a build _starts_ and on which builder, so we can find the rbuild log * missing tests: variation in kernel and date and cpu type and user group ** prebuilder does group variation like this: https://anonscm.debian.org/cgit/reproducible/misc.git/tree/prebuilder/pbuilderhooks/A02_user ** different cpu type: Opteron_G3 AMD Opteron 23xx (Gen 3 Class Opteron) is the most powerful one that's different to current Opteron_G4 * a test VM using this build-slave design has been setup: ** 9 cores, 36gb ram, 80gb disk, 50gb /, 30gb /srv/kvm ** runs squid3 + kvm *** though squid is not configured yet ** kvm guest with different cpu, 32gb ram, 8 cores, 30gb / ** date changed in /etc/rc.local to today plus two days, one month and one year and hoping that the apt-signing key is still valid ** runs pbuilder inside the kmv guest * enable people to upload test packages, to be built in jenkins: ---- h01ger: another wild future request by me: allowing us to upload something and let jenkins test it. rationale: I sent (another) patch for debian-keyring, to fix a timestamp issue in debian control files (due to not_using_dh-builddeb), but there is also a umask issue. I don't want to bother me to setup the very same things jenkins tests locally (I already did too much in this regards, imho), but really people can't tests everything jenkins tests. mapreri: please add the feature request to the todo. i'm thinking now that it maybe should just be a jenkins job not integrated into the rp.d.n webui, but... maybe we find a nice way to do it h01ger: I'm instead thinking about a repo defining a reproducible-specific suite or something on that line, that integrates well with the current setup. but this is really something wild. well, and everybody in debian-keyring from sid can uplood? :) that would be wonderful. ---- * debootstrap ** add the test (something weekly or so) ** https://wiki.debian.org/ReproducibleInstalls * coreboot ** add more variations: domain+hostname, uid+gid, USER, UTS namespace ** split into two jobs, one to build the toolchain, the other to test, keep the git repo and update it. run both jobs weekly. *** call script with "init" or "run-tests" *** not sure how to best keep the git repo and the toolchain (not on tmpfs) while building in tmpfs... * openwrt ---- auschecken / updaten macht man ja einfach global auf dem tree mit git wenn man einmal make defconfig ausgeführt hat, gibt's paket und target metadaten in tmp/.packageinfo und tmp/.targetinfo man kann ne .config mit der target auswahl und CONFIG_ALL=y erstellen und mit make defconfig auffüllen macht wahrscheinlich sinn vom prozess her das bauen von paketen und das bauen der toolchain zusammen laufen zu lassen weil openwrt das ja alles in einem durchgang macht reicht ja erstmal, mit dem basis paketsatz anzufangen das wird für ein target deutlich weniger als 12h brauchen default config ohne CONFIG_ALL dauert bei meinem laptop in der regel <30 min mit CONFIG_ALL vielleicht 2-3 stunden oder so ---- * remote scheduling: ---- | h01ger: I have a slave configured, named buildbot.pixelminers. remote root is /home/jenkins/pseudo-hosts/buildbot.pixelminers.net and launch command is ssh localhost ~/pseudo-hosts/jenkins-tools/slaves/linux/start-slave.sh https://www.palfrader.org/volatile/2015-06-06-sP55pjoGcN8/screenshot.png https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux - job: name: tor-ci-freebsd-amd64-master project-type: freestyle node: freebsd-amd64 builders: - shell: '~/jenkins-tools/slaves/other/build-wrapper' https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/other/build-wrapper https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/other/tor-ci-freebsd-amd64-master/build ---- === qa.debian.org* * udd-versionskew: explain jobs in README * udd-versionskew: also provide arch-relative version numbers in output too === d-i_manual* * d-i_check_jobs.sh: check for removed manuals (but with existing jobs) missing * svn:trunk/manual/po triggers the full build, should trigger language specific builds. * svn:trunk/manual is all thats needed, not whole svn:trunk === d-i_build* * d-i_check_jobs.sh: check for removed package (but with existing jobs) missing * build packages using jenkins-debian-glue and not with the custom scripts used today? * run scripts/digress/ ? * bubulle wrote: "Another interesting target would be d-i builds *including non uploaded packages* (something like "d-i from git repositories" images). That would in some way require to create a quite specific image, with all udebs (while netboot only has udebs needed before one gets a working network setup). === chroot-installation_* * use schroot for chroot-installation, stop using plain chroot everywhere * add alternative tests with aptitude and possible apt * split etc/schroot/default * inform debian-devel@l.d.o or -qa@? * warn about transitional packages installed (on non-upgrades only) * install all the tasks "instead", thats rather easy nowadays as all task packages are called "task*". ** make sure this includes blends === g-i-installation_* Development of these tests has stopped. In future the 'lvc*' tests should replace them. These small changes are probably still worth doing anyway: * g-i: replace '--' with '---' as param delimiter. see #776763 / 5df5b95908 in d-e-c * download .isos once in central place ** /var/lib/jenkins/jobs/g-i-installation_*/workspace/*iso needs 53GB currently, it could be 30 less * g-i_presentation: use preseeding files on jenkins.d.n and not hands.com * turn job-cfg/g-i.yaml into .yaml.py The following ideas should really only be implemented for the new 'lvc*' tests.... (but are kept here for now) * pick LANG from predefined list at random - if last build was not successful or unstable fall back to English ** these jobs would not need to do an install, just booting them in rescue mode is probably enough * for edu mainservers running as servers for workstations etc: "d-i partman-auto/choose_recipe select atomic" to be able to use smaller disk images ** same usecase: -monitor none -nographic -serial stdio == Further ideas... === rebuild sid completly on demand * nthykier wants to be able to rebuild all of sid to test how changes to eg lintian, debhelper, cdbs, gcc affect the archive: * h01ger> | nthykier: so a.) rebuild everything from sid plus custom repo. b.) option to only rebuild a subset, like all rdepends or all packages build-depending on something * h01ger> | and c.) only build once, not continously and d.) enable more cores+ram on demand to build faster === Test them all * build packages from all team repos on alioth with jenkins-debian-glue on team request (eg, via a .txt file in a git.repo) for specific branches (which shall also be automated, eg. to be able to only have squeeze+sid branches build, but not all other branches.) == Debian Packaging related This setup should come as a Debian source package... * /usr/sbin/jenkins.debian.net-setup needs to be written * what update-j.d.n.sh does, needs to be put elsewhere... * debian/copyright is incorrect about some licenses: ** the profitbricks+debian+jenkins logos ** the preseeding files ** ./feature/ is gpl3 // vim: set filetype=asciidoc: