ToDo for jenkins.debian.net =========================== :Author: Holger Levsen :Authorinitials: holger :EMail: holger@layer-acht.org :Status: working, in progress :lang: en :Doctype: article :Licence: GPLv2 == About jenkins.debian.net See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian.net"] for a general description of the setup. Below is the current TODO list, which is long and probably incomplete too. The links:https://jenkins.debian.net/userContent/contributing.html[the preferred form of contributions] are patches via pull requests. == Fix user submitted bugs * There are link:https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=jenkins;users=qa.debian.org%40packages.debian.org["bugs filed against the pseudopackage 'qa.debian.org' with usertag 'jenkins'"] in the BTS which would be nice to be fixed soon, as some people actually care. == General ToDo * replace amd64 in scripts with $HOSTARCH * extend /etc/rc.local to do cleanup of lockfiles * explain in README how to write jobs, eg which pathes are on tmpfs === proper backup * gpg encrypted to some keys * run on alioth or paradis * '/var/lib/jenkins/jobs' (the results - the configs are in .git) * '/var/lib/munin' * '/var/log' * '/root/' (contains etckeeper.git) * '/var/lib/jenkins/reproducible.db' (is backed up manually) * '/srv/jenkins.debian.net-scm-sync.git' (is backed up manually) * '/var/lib/jenkins/plugins/*.jpi' (can be derived from jdn-scm-sync.git) * '/srv/jenkins.debian.net-scm-sync.git' * '/etc/.git' and '/etc' * postpone til we run on .debian.org? === TODO for testing stretch Most jobs have been converted, a few are left to do: * add g-i tests for stretch * add stretch live-builds * do lvc for stretch too * mention stretch in README where appropriate === move this setup to jenkins.d.o The plan is to run a jenkins.d.o host, which is maintained by DSA, but we are maintaining jenkins on it (so we can install any plugins we like etc). then we also setup several jenkins slaves, probably/maybe also maintained by DSA (so we get them into their munin), but on which we can use sudo as we need it. (or maybe not dsa-maintained slaves, so that we can use sudo as we need, for the price of not being in DSAs munin.) ==== next steps for jenkins.d.o migration * weasel/h01ger: install jenkins.deb ** also create jenkins users in jenkins (KISS) * h01ger: get slaves: wishlist for starting: 3 slaves, 8 cores, 32gb ram, 150gb hd space if we dont need squid3 on them, 200gb if we do. ** install slaves - how (to automate)? * install jenkins-job-builder ** needs proper package * ... (to be planned, see below) * update DNS to point jenkins.d.o to jerea.d.o * ... (to be planned, see below) ==== unsorted notes for jenkins.d.o migration * chroot jobs should use real schroot sessions, and not just use schroot as poor chroot(8) replacement. some links: ** https://anonscm.debian.org/cgit/mirror/dsa-puppet.git/tree/modules/schroot ** https://anonscm.debian.org/cgit/mirror/dsa-puppet.git/tree/modules/porterbox/files/dd-schroot-cmd ** https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/build-wrapper * sudoers.d/jenkins: ** not suitable for jenkins.d.o, thus we will run all tests on slaves, where DSA doesnt care what we do * upgrade to jessie, software used which is not in jessie / available as jenkins plugin: ** jenkins.deb *** DSA prefers if we could use jenkins from jessie-backports *** 2nd option: own repo, only contains jenkins.deb *** 3rd option: use upstreams repo ** jenkins-job-builder probably needs to be more properly packaged *** could be installed locally in jenkins home ** livescreenshot plugin (we use a patched version) *** jenkins maintaince probably is best done by jenkins users (as opposed to DSA) so that's up to us * munin monitoring of the slaves ** DSA munin configuration is auto generated by puppet, so the slaves should become .d.o hosts too, to be included * the existing jenkins.d.o host needs to be renamed to something else (thats "just work" to do but not a major obstacle) === To be done once jenkins.d.n runs jessie * replace with bin/setsid.py workaround with setsid from the util-linux package from jessie * bin/g-i-installation: use lvcreate without --virtualsize * check if the sudo workaround in bin/g-i-installation is still needed: 'guestmount -o uid=$(id -u) -o gid=$(id -g)' would be nicer, but it doesnt work: as root, the files seem to belong to jenkins, but as jenkins they cannot be accessed. * upload pbuilder to jessie-bpo once its ready and install pbuilder from there * install botch from jessie-backports (and remove botch from the reproducible-unstable schroot) === To be done once bugs are fixed * link:https://bugs.debian.org/767260[#767260] workaround in bin/d-i_build.sh (console-setup doesn't support parallel build) * link:https://bugs.debian.org/767032[#767032] manual fix in etc/munin/plugins/munin_stats * link:https://bugs.debian.org/767100[#767100] work in progress in etc/munin/plugins/cpu * link:https://bugs.debian.org/767018[#767018] work in progress in etc/munin/plugins/iostat_ios * link:https://bugs.debian.org/774685[#774685] workaround in bin/reproducible_create_meta_pkg_sets.sh === jenkins-job-builder related * use jessie version plus h01ger's patches from kali * change of syntax: ---- properties: - priority-sorter: priority: 150 ---- * this seems to be helpful: http://en.wikipedia.org/wiki/YAML#References (pyyaml which jenkins-job-builder uses supports them) * cleanup h01ger's patches (eg add documentation) and send pull requests on github: ** publisher:logparse ** publisher:htmlpublisher ** svn:scm upstreamed at https://review.openstack.org/#/c/192095/ ** wrappers:live-screenshot upstreamed at https://review.openstack.org/#/c/191708/ ** image-gallery: https://review.openstack.org/#/c/175747/ superseeds h01gers patch: https://review.openstack.org/#/c/191950/ ** sidebar: upstreamed at https://review.openstack.org/#/c/191585/ === livescreenshot plugin * publish forked livescreenshot plugin and send pull request for h01ger's bugfix ** see ssh://git.debian.org/git/users/holger/livescreenshot-plugin.git and 0b407b70025 there == lvc, work in progress, just started * put this on debian isos too: config/chroot_local-includes/lib/live/config/9999-autotest * add another (smaller) test: download+run torbrowser daily * re-read the docs! ** http://live.debian.net/manual/stable/html/live-manual.en.html#321 * generate feature files from templates? to cope with sub-products? -> no. detect desktop type and set variables accordingly -> simpler: pass an environment variable with the type * get iso * tables for looping through features: see tails/iuk.git/features/download_target_file/Download_Target_File.feature * to debug cucumber: --verbose --backtrace --expand * drop / remove * can probably go: dhcp.rb firewall_leaks.rb dhcp.feature firewall_leaks.feature * more occurances of "the computer boots Tails" * @source (only keep product tests) * disabled stuff in common_steps.rb ** #if @vm.execute("service tor status").success? * "I set sudo password" not needed for debianlive nor debian(edu): ** #@screen.wait("TailsGreeterAdminPassword.png", 20) * $misc_files_dir needed? * def sort_isos_by_creation_date Dir.glob("#{Dir.pwd}/*.iso").sort_by {|f| tails_iso_creation_date(f)} -> useless for us, purpose is to automatically select the latest iso if none is given * search case-in-sensitive for tails+tor+amnesia * put in update_jdn.sh: ---- addgroup tcpdump dpkg-statoverride --update --add root tcpdump 754 /usr/sbin/tcpdump setcap CAP_NET_RAW+eip /usr/sbin/tcpdump adduser $USER tcpdump adduser $USER libvirt adduser $USER libvirt-qemu ---- == Improve existing tests === reproducible * higher prio: ** fix https://jenkins.debian.net/munin/debian.net/jenkins.debian.net/jenkins_builds.html which is broken since jessie upgrade * lesser prio ** html_graphs.sh rewrite: split out html_pkg_sets.py first, thats the easiest and the biggest speed gain ** more graphs: *** graph average build duration by day *** graph packages in testing+unstable which need to be fixed ** reproducible_create_meta_pkg_sets uses schroot created by dpkg_setup_schroot_jessie job (outside of reproducible job space...) ** "fork" etc/schroot/default into etc/schroot/reproducible ** repo-comparison: check for binaries without source ** move "untested" field in stats table too? (as in csv output...) ** new page: packages which are orphaned but have a reproducible usertagged patch ** a reproducible_log_grep_by_sql.(py|sh) would be nice, to only grep in packages with a certain status (build in the last X days) ** replace submit form by one without javascript (maybe with more url rewriting) ** html_indexes.py creates /index_notify.html three times, even though one is enough. ** when a package is automatically rescheduled because of the mirror was updated between the two tests, there will be three rbuild logs in one. thats confusing, the first one should be dropped. ** reproducible_blacklist.sh should delete rbuild logs and debbindiff output too ** adopt usertag script from pkg-apparmor to notify us about new usertagged bugs automatically ** fix apache ssl configuration as hinted by eg https://sslcheck.globalsign.com/de/sslcheck?host=jenkins.debian.net#46.16.73.183 ** create a symbol for pending bugs or use a different color to indicate them * notes related ** #786396: classify issue by "toolchain" or "package" fix needed ** new page with annoted packages without categorized issues ** new page with notes that doesnt make sense: a.) packages which are reproducible but should not, packages that build but shouldn't, etc. *** aint that covered by reproducible_breakages.py already? no. * pkg sets related: ** fix essential set: currently it only has the ones explicitly marked Essential:yes; they and their dependencies make up the full "essential closure set" (sometimes also called pseudo-essential) ** replace bin/reproducible_installed_on_debian.org with a proper data provider from DSA, eg https://anonscm.debian.org/cgit/mirror/debian.org.git/plain/debian/control * missing tests: ** variation in kernel ** variation in date ** prebuilder does (user) group variation like this: https://anonscm.debian.org/cgit/reproducible/misc.git/tree/prebuilder/pbuilderhooks/A02_user ** different cpu type: Opteron_G3 AMD Opteron 23xx (Gen 3 Class Opteron) is the most powerful one that's different to current Opteron_G4 ** variation of $TERM and $COLUMN (and maybe $LINES), unset in the first run, set to "linux" and "77" (and maybe "42") in the 2nd run. maybe vary $SHELL too. * a test VM using this build-slave design has been setup: ** 9 cores, 36gb ram, 80gb disk, 50gb /, 30gb /srv/kvm ** runs squid3 + kvm *** though squid is not configured yet ** kvm guest with different cpu, 32gb ram, 8 cores, 30gb / ** date changed in /etc/rc.local to today plus two days, one month and one year and hoping that the apt-signing key is still valid ** runs pbuilder inside the kmv guest * enable people to upload test packages, to be built in jenkins: ---- <mapreri> h01ger: another wild future request by me: allowing us to upload something and let jenkins test it. rationale: I sent (another) patch for debian-keyring, to fix a timestamp issue in debian control files (due to not_using_dh-builddeb), but there is also a umask issue. I don't want to bother me to setup the very same things jenkins tests locally (I already did too much in this regards, imho), but really people can't tests everything <mapreri> jenkins tests. <h01ger> mapreri: please add the feature request to the todo. i'm thinking now that it maybe should just be a jenkins job not integrated into the rp.d.n webui, but... maybe we find a nice way to do it <mapreri> h01ger: I'm instead thinking about a repo defining a reproducible-specific suite or something on that line, that integrates well with the current setup. but this is really something wild. <h01ger> well, and everybody in debian-keyring from sid can uplood? :) <mapreri> that would be wonderful. ---- ==== reproducible Debian installation * see https://wiki.debian.org/ReproducibleInstalls * add the test (something weekly or so) ==== reproducible coreboot * add more variations: domain+hostname, uid+gid, USER, UTS namespace * build the docs? * also build with payloads. x86 use seabios as default, arm boards dont have a default. grub is another payload. and these: bayou coreinfo external filo libpayload nvramcui - and: ** CONFIG_PAYLOAD_NONE=y ** CONFIG_PAYLOAD_ELF is not set ** CONFIG_PAYLOAD_LINUX is not set ** CONFIG_PAYLOAD_SEABIOS is not set ** CONFIG_PAYLOAD_FILO is not set ** CONFIG_PAYLOAD_GRUB2 is not set ** CONFIG_PAYLOAD_TIANOCORE is not set * libreboot ships images, verify those? * explain status in plain english ==== reproducible openwrt * add credit for logo/artwork * build more archs (http://downloads.openwrt.org/chaos_calmer/15.05-rc1/ lists many to choose from) * build all packages? (set CONFIG_ALL=y and run 'make defconfig') ** just build some first... * file dbd bug about unable to inspect these .bin files * file dbd bug about crashing on certain squashfs files * explain status in plain english ==== reproducible netbsd * announce on their list * explain status in plain english ** MKREPRO is set to "yes" ==== reproducible fedora * use mock to create a fedora chroot to build in * start with building a single package * then build the full base system (100-500 packages) ==== reproducible freebsd * needs a freebsd system to build * first build world, later build ports * use this setup as the first test bed for remote scheduling: ==== remote scheduling: ---- <weasel> | h01ger: I have a slave configured, named buildbot.pixelminers. <weasel> remote root is /home/jenkins/pseudo-hosts/buildbot.pixelminers.net <weasel> and launch command is ssh localhost ~/pseudo-hosts/jenkins-tools/slaves/linux/start-slave.sh <weasel> https://www.palfrader.org/volatile/2015-06-06-sP55pjoGcN8/screenshot.png <weasel> https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux <weasel> - job: <weasel> name: tor-ci-freebsd-amd64-master <weasel> project-type: freestyle <weasel> node: freebsd-amd64 <weasel> builders: <weasel> - shell: '~/jenkins-tools/slaves/other/build-wrapper' <weasel> https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/other/build-wrapper <weasel> https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/other/tor-ci-freebsd-amd64-master/build ---- === qa.debian.org* * udd-versionskew: explain jobs in README * udd-versionskew: also provide arch-relative version numbers in output too === d-i_manual* * d-i_check_jobs.sh: check for removed manuals (but with existing jobs) missing * svn:trunk/manual/po triggers the full build, should trigger language specific builds. * svn:trunk/manual is all thats needed, not whole svn:trunk === d-i_build* * d-i_check_jobs.sh: check for removed package (but with existing jobs) missing * build packages using jenkins-debian-glue and not with the custom scripts used today? * run scripts/digress/ ? * bubulle wrote: "Another interesting target would be d-i builds *including non uploaded packages* (something like "d-i from git repositories" images). That would in some way require to create a quite specific image, with all udebs (while netboot only has udebs needed before one gets a working network setup). === chroot-installation_* * use schroot for chroot-installation, stop using plain chroot everywhere * add alternative tests with aptitude and possible apt * split etc/schroot/default * inform debian-devel@l.d.o or -qa@? * warn about transitional packages installed (on non-upgrades only) * install all the tasks "instead", thats rather easy nowadays as all task packages are called "task*". ** make sure this includes blends === g-i-installation_* Development of these tests has stopped. In future the 'lvc*' tests should replace them. These small changes are probably still worth doing anyway: * g-i: replace '--' with '---' as param delimiter. see #776763 / 5df5b95908 in d-e-c * download .isos once in central place ** /var/lib/jenkins/jobs/g-i-installation_*/workspace/*iso needs 53GB currently, it could be 30 less * g-i_presentation: use preseeding files on jenkins.d.n and not hands.com * turn job-cfg/g-i.yaml into .yaml.py The following ideas should really only be implemented for the new 'lvc*' tests.... (but are kept here for now) * pick LANG from predefined list at random - if last build was not successful or unstable fall back to English ** these jobs would not need to do an install, just booting them in rescue mode is probably enough * for edu mainservers running as servers for workstations etc: "d-i partman-auto/choose_recipe select atomic" to be able to use smaller disk images ** same usecase: -monitor none -nographic -serial stdio == Further ideas... === rebuild sid completly on demand * nthykier wants to be able to rebuild all of sid to test how changes to eg lintian, debhelper, cdbs, gcc affect the archive: * h01ger> | nthykier: so a.) rebuild everything from sid plus custom repo. b.) option to only rebuild a subset, like all rdepends or all packages build-depending on something * h01ger> | and c.) only build once, not continously and d.) enable more cores+ram on demand to build faster === Test them all * build packages from all team repos on alioth with jenkins-debian-glue on team request (eg, via a .txt file in a git.repo) for specific branches (which shall also be automated, eg. to be able to only have squeeze+sid branches build, but not all other branches.) == Debian Packaging related This setup should come as a Debian source package... * /usr/sbin/jenkins.debian.net-setup needs to be written * what update-j.d.n.sh does, needs to be put elsewhere... * debian/copyright is incorrect about some licenses: ** the profitbricks+debian+jenkins logos ** the preseeding files ** ./feature/ is gpl3 // vim: set filetype=asciidoc: