diff options
| author | Holger Levsen <holger@layer-acht.org> | 2017-06-28 11:43:40 +0200 |
|---|---|---|
| committer | Holger Levsen <holger@layer-acht.org> | 2017-06-28 11:43:40 +0200 |
| commit | 77c6891eb21c8fa5cbffe22f75d9e54fbe10b797 (patch) | |
| tree | 1de4ff9f41718635890cc0ef79cb9516d4a21fc6 | |
| parent | 1542df26bd0d4ac34df04528bb6cb4eda8388002 (diff) | |
| download | jenkins.debian.net-77c6891eb21c8fa5cbffe22f75d9e54fbe10b797.tar.xz | |
reproducible Debian: check for (known) zombie jobs and report them
Signed-off-by: Holger Levsen <holger@layer-acht.org>
| -rwxr-xr-x | bin/reproducible_maintenance.sh | 1 | ||||
| -rwxr-xr-x | bin/reproducible_node_health_check.sh | 19 | ||||
| -rw-r--r-- | logparse/reproducible.rules | 1 |
3 files changed, 21 insertions, 0 deletions
diff --git a/bin/reproducible_maintenance.sh b/bin/reproducible_maintenance.sh index 4543441d..d93676c9 100755 --- a/bin/reproducible_maintenance.sh +++ b/bin/reproducible_maintenance.sh @@ -336,6 +336,7 @@ if [ "$HOSTNAME" = "$MAINNODE" ] ; then echo "Removed $OLDSTUFF jenkins html logs." echo fi + fi # find+terminate processes which should not be there diff --git a/bin/reproducible_node_health_check.sh b/bin/reproducible_node_health_check.sh index 189bc984..16466822 100755 --- a/bin/reproducible_node_health_check.sh +++ b/bin/reproducible_node_health_check.sh @@ -137,6 +137,25 @@ if [ -z "$HAVEGED" ] ; then fi # +# checks only for the main node +# +if [ "$HOSTNAME" = "$MAINNODE" ] ; then + # + # sometimes deleted jobs come back as zombies + # and we dont know why and when that happens, + # so just report those zombies here. + # + ZOMBIES=$(ls -1d var/lib/jenkins/jobs/ | egrep '(reproducible_builder_amd64|reproducible_builder_i386|reproducible_builder_armhf|reproducible_builder_arm64|chroot-installation_wheezy)') + if [ ! -z "$ZOMBIES" ] ; then + echo "Warning, rise of the jenkins job zombies has started again, these jobs should not exist:" + echo -e "$ZOMBIES" + DIRTY=TRUE + echo + fi +fi + + +# # finally # if ! $DIRTY ; then diff --git a/logparse/reproducible.rules b/logparse/reproducible.rules index 39faf2e1..ef19fef9 100644 --- a/logparse/reproducible.rules +++ b/logparse/reproducible.rules @@ -24,4 +24,5 @@ warning /Warning: /.../shm.+please tell the jenkins admins to fix this./ warning /Warning, today is the wrong future:.+/ warning /Warning: curl http:..www.debian.org failed, probably the proxy is down for.+/ warning /Warning, more than one kernel in.+/ +warning /Warning, rise of the jenkins job zombies.+/ |