update documentation and todo

4 files changed, 79 insertions, 17 deletions

diff --git a/INSTALL b/INSTALL
index 63374761..ea87cbcd 100644
--- a/INSTALL
+++ b/INSTALL
@@ -60,7 +60,7 @@ cd jenkins.debian.net
 ./update_jdn.sh
 ----
 
-'jenkins-job-builder' is the only software which is not covered by this script.
+'jenkins-job-builder' and (currently) 'kgb-client' is the only software which is not covered by this script.
 
 === Installing jenkins-job-builder
 
@@ -76,13 +76,23 @@ python setup.py install
 
 Though actually https://github.com/h01ger/jenkins-job-builder is used atm.
 
-=== Using jenkins-job-builder
+==== Using jenkins-job-builder
 
-'update_jdn.sh' takes care of this too, so this is just for quick documentation. The real documentation is at link:http://ci.openstack.org/jenkins-job-builder/
+'update_jdn.sh' takes care of this too, so this is just for quick documentation. The real documentation is at http://ci.openstack.org/jenkins-job-builder/
 
 * run `cd /srv/jenkins/job-cfg ; jenkins-jobs update .` to update jobs
 * for testing: `cd /srv/jenkins/job-cfg ; rm test-output/ -r ; mkdir test-output ; jenkins-jobs test . -o test-output/ ; ls test-output/`
 
+=== Installing kgb-client
+
+Rebuild the 'kgb-bot' source package from sid on whezzy and then install the 'kbg-client' package. Configuration is done through 'update_jdn.sh'.
+
+==== IRC notification configuration
+
+The jenkins jobs are configured to send email to 'jenkins+$IRC_CHANNEL' (like 'jenkins+debian-qa'), this is parsed by a script ('/srv/jenkins/bin/email2irc.sh') run through 'procmail' which then in turn notifies 'kgb-client', which notfies 'kgb-server'(s) on the internet, which are the bots notifying the IRC channels. 'kbg-client' needs to be '>= 1.24'.
+
+The Jenkins EMail notification plugin is used as its state tracking is best (and the Jenkins IRC plugin is way too powerful).
+
 === munin
 
 link://munin-monitoring[Munin] is installed, running with link:http://jenkins.debian.net/munin/[jenkins plugins] not yet published elsewhere.
diff --git a/README b/README
index 8167e173..c9c0130c 100644
--- a/README
+++ b/README
@@ -68,7 +68,7 @@ Each job pdebuilds the master branch of its git repo on every git push in a sid
 * d_i_manual - builds the full installation-guide package with pdebuild in sid on every commit to svn://svn.debian.org/svn/d-i/ matching these patterns: 
 '/trunk/manual/debian/.\*', '/trunk/manual/po/.\*', '/trunk/manual/doc/.\*' and '/trunk/manual/scripts/.\*'
 ** while this job are triggered on commits, the SCM is only polled every 30min to see if there are new commits.
-* d_i_manual_* - builds a language (on wheezy) on every commit of svn/trunk/manual/$LANG with "make languages=$LANG architectures=amd64 formats=html"
+* d_i_manual_*_html - builds a language (on wheezy) on every commit of svn/trunk/manual/$LANG with "make languages=$LANG architectures=amd64 formats=html". On successful build, d_i_manual_$lang_pdf is triggered.
 ** while these jobs are triggered on commits, the SCM is only polled every 30min to see if there are new commits.
 * d_i_parse_build_logs - parses logs from http://d-i.debian.org/daily-images/build-logs.html daily, to give them a bit more exposure.
 ** this job is run daily.
diff --git a/TODO b/TODO
index 243ace15..87b14bb9 100644
--- a/TODO
+++ b/TODO
@@ -14,11 +14,25 @@ See link:http://jenkins.debian.net/userContent/about.html["about jenkins.debian.
 
 == General ToDo
 
+* switch git:// and svn:// urls to http:// and thus use proxy. increase polling frequency to 2min.
 * general housekeeping job:
 ** run calamaris, visitors etc
 ** check for stale processes
 ** check for apt-upgrades
 ** check for jobs forgetting their triggers
+* proper git repo url, outside users/holger
+** same for /srv/jenkins.debian.net-scm-sync.git
+* chroot-tests: only trigger (=really run) jobs if available+used packages have changed (save packages in db and compare)
+* update jdk
+* fully automate backup and full backup /var/lib/jenkins/jobs and /var/lib/munin and /var/log too
+* try awstats
+
+=== Bugs
+
+* html_publisher doesn't make the results of d-i_manual_* available
+
+=== jenkins-job-builder related
+
 * support for gitweb in jobs 
 ** needs clean merge of my patches to jenkins-job-builder first...
 * get rid of some reduncacy in job-cfg/*.yaml 
@@ -28,28 +42,28 @@ See link:http://jenkins.debian.net/userContent/about.html["about jenkins.debian.
 ** svn:scm
 ** properties: sidebar
 * file a bug about jenkins/jenkins-job-builder sometimes "forgetting" about triggers
-* mail notifications (to where?)
-* proper git repo url, outside users/holger
-** same for /srv/jenkins.debian.net-scm-sync.git
-* only trigger (=really run) jobs if available+used packages have changed (save packages in db and compare)
-* update jdk
-* fully automate backup and full backup /var/lib/jenkins/jobs and /var/lib/munin and /var/log too
-* try awstats
 
 == More tests to be run
 
 === d-i_manual*
 
-* html_publisher doesn't make the results of d-i_manual_* available
-* svn:trunk:packages/po/* triggers nothing
-* notify #debian-boot about jenkins jobs results
+* svn:trunk/manual/po triggers the full build, should trigger language specific builds.
+----
+As documented in manual/doc/translations_po.txt:
+        Use the following commands:
+          1. ./scripts/merge_xml en
+          2. ./scripts/update_pot
+          3. ./scripts/update_po <your language>
+          4. ./scripts/revert_pot
+          5. ./scripts/create_xml <your language>
+----
+* inform translators (and debian-boot for the package)
 
 === d-i_build*
 
 * run scripts/digress/
 * bubulle wrote: "Another interesting target would be d-i builds *including non uploaded packages* (something like "d-i from git repositories" images). That would in some way require to create a quite specific image, with all udebs (while netboot only has udebs needed before one gets a working network setup).
-* notify #debian-boot about jenkins jobs results
-* run scripts/digress/
+* inform debian-boot@ ?
 
 === chroot-test_*
 
@@ -89,3 +103,41 @@ Help explaining how to test tasks (ie all the language tasks) very much welcome!
 * finding packages with autopkgtests: http://lists.debian.org/debian-qa/2012/11/msg00012.html
 ** this only lists a few packages, so a very first test could be to compare this list against this: `wget -q http://http.debian.net/debian/dists/unstable/main/Contents-source.gz -O- | zgrep -m1 -E '^debian/tests/control\s'`, make a job out of it and make it UNSTABLE if the diff is non-zero. And as a second step, run those tests...
 * also see http://dep.debian.net/deps/dep8/
+* also from that thread on debian-qa@l.d.o:
+----
+Subject: Re: automated tests or builds for d-i
+From: Martin Pitt <mpitt@debian.org>
+
+  Stefano Zacchiroli [2012-11-04 14:51 +0100]:
+> AFAIK in Ubuntu they've glue code that hooks autopkgtest tests into
+> jenkins. I don't know where the code is or how Ubuntu-specific it is,
+> but it is worth investigating. It'd be awesome to make it as
+> distro-agnostic as possible, enabling all derivatives to run their own
+> autopkgtest tests: it'd be a nice way to spot regressions introduced by
+> distro-specific patches and more.
+
+Indeed, our goal is to get a lot more of them. We currently have some
+20 [1], but at the last UDS we had a little competition which got us
+another 11 [2], which we'll upload and forward to Debian in the next
+days.
+
+Our scripts for creating VMs, running the tests in those, and the
+Jenkins integration are in "bzr branch lp:auto-package-testing", which
+you can browse at [3]. In the doc/ directory there is documentation
+how the Jenkins integration works. I can't say much about it I'm
+afraid, as I have never touched a Jenkins job so far; this has been
+created by Jean-Baptiste Lallement (in CC:); he's on holiday for the
+next week, though.
+
+Please let me know if this is useful. I can poke at the jobs in our
+actual Jenkins instance (jenkins.qa.u.c. is just a readonly public
+mirror) if you need to know something specific, but we try to avoid
+having any manual job configuration; these are all autogenerated
+through scripts through python-jenkins.
+
+Martin
+
+[1] https://jenkins.qa.ubuntu.com/view/Quantal/view/AutoPkg%20Test/
+[2] https://bugs.launchpad.net/ubuntu/+bugs?field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.tag=udstestcompetition
+[3] http://bazaar.launchpad.net/~auto-package-testing-dev/auto-package-testing/trunk
+----
diff --git a/etc/shorewall/rules b/etc/shorewall/rules
index db08726c..4b95dd6f 100644
--- a/etc/shorewall/rules
+++ b/etc/shorewall/rules
@@ -26,7 +26,7 @@ Ping(DROP)	net		$FW
 
 ACCEPT		$FW		net		icmp
 
-# http and ssh are allowed
+# incoming http and ssh are allowed
 ACCEPT    net       $FW             tcp          80
 ACCEPT    net       $FW             tcp          22