|
2016-03-01
§
|
| 16:41 |
<hashar> |
Restarted Jenkins |
[releng] |
| 16:32 |
<hashar> |
Bunch of Jenkins job got stall because I have killed threads in Jenkins to unblock integration-slave-trusty-1003 :-( |
[releng] |
| 12:14 |
<hashar> |
integration-slave-trusty-1003 is back online |
[releng] |
| 12:13 |
<hashar> |
Might have killed the proper Jenkins thread to unlock integration-slave-trusty-1003 |
[releng] |
| 12:03 |
<hashar> |
Jenkins can not pool back integration-slave-trusty-1003 Jenkins master has a bunch of blocking threads pilling up with hudson.plugins.sshslaves.SSHLauncher.afterDisconnect() locked somehow |
[releng] |
| 11:41 |
<hashar> |
Rebooting integration-slave-trusty-1003 (does not reply to salt / ssh) |
[releng] |
| 10:34 |
<hashar> |
Image ci-jessie-wikimedia-1456827861 in wmflabs-eqiad is ready |
[releng] |
| 10:24 |
<hashar> |
Refreshing Nodepool snapshot instances |
[releng] |
| 10:22 |
<hashar> |
Refreshing Nodepool base image to speed instances boot time (dropping open-iscsi package https://gerrit.wikimedia.org/r/#/c/273973/ ) |
[releng] |
|
2016-02-26
§
|
| 15:14 |
<jzerebecki> |
salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" T128191 |
[releng] |
| 15:14 |
<jzerebecki> |
salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" |
[releng] |
| 14:44 |
<hashar> |
(since it started, dont be that scared!) |
[releng] |
| 14:44 |
<hashar> |
Nodepool has triggered 40 000 instances |
[releng] |
| 11:53 |
<hashar> |
Restarted memcached on deployment-memc02 T128177 |
[releng] |
| 11:53 |
<hashar> |
memcached process on deployment-memc02 seems to have a nice leak of socket usages (from lost) and plainly refuse connections (bunch of CLOSE_WAIT) T128177 |
[releng] |
| 11:53 |
<hashar> |
memcached process on deployment-memc02 seems to have a nice leak of socket usages (from lost) and plainly refuse connections (bunch of CLOSE_WAIT) |
[releng] |
| 11:40 |
<hashar> |
deployment-memc04 find /etc/apt -name '*proxy' -delete (prevented apt-get update) |
[releng] |
| 11:26 |
<hashar> |
beta: salt -v '*' cmd.run 'apt-get -y install ruby-msgpack' . I am tired of seeing puppet debug messages: "Debug: Failed to load library 'msgpack' for feature 'msgpack'" |
[releng] |
| 11:24 |
<hashar> |
puppet keep restarting nutcracker apparently T128177 |
[releng] |
| 11:20 |
<hashar> |
Memcached error for key "enwiki:flow_workflow%3Av2%3Apk:63dc3cf6a7184c32477496d63c173f9c:4.8" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY |
[releng] |
|
2016-02-25
§
|
| 22:38 |
<hashar> |
beta: maybe deployment-jobunner01 is processing jobs a bit faster now. Seems like hhvm went wild |
[releng] |
| 22:23 |
<hashar> |
beta: jobrunner01 had apache/hhvm killed somehow .... Blame me |
[releng] |
| 21:56 |
<hashar> |
beta: stopped jobchron / jobrunner on deployment-jobrunner01 and restarting them by running puppet |
[releng] |
| 21:49 |
<hashar> |
beta did a git-deploy of jobrunner/jobrunner hoping to fix puppet run on deployment-jobrunner01 and apparently it did! T126846 |
[releng] |
| 11:21 |
<hashar> |
deleting workspace /mnt/jenkins-workspace/workspace/browsertests-Wikidata-WikidataTests-linux-firefox-sauce on slave-trusty-1015 |
[releng] |
| 10:08 |
<hashar> |
Jenkins upgraded T128006 |
[releng] |
| 01:44 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/273170 |
[releng] |
| 01:39 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/272955 (undeployed) and https://gerrit.wikimedia.org/r/273136 |
[releng] |
| 01:37 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/273136 |
[releng] |
| 00:31 |
<thcipriani> |
running puppet on beta to update scap to latest packaged version: sudo salt -b '10%' -G 'deployment_target:scap/scap' cmd.run 'puppet agent -t' |
[releng] |
| 00:20 |
<thcipriani> |
deployment-tin not accepting jobs for some time, ran through https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update, is back now |
[releng] |