|
2016-02-09
ยง
|
| 16:53 |
<bblack> |
rebooting cp1008/pinkunicorn for 4.4 kernel |
[production] |
| 16:34 |
<jynus> |
reimage db2012 |
[production] |
| 16:30 |
<thcipriani@mira> |
Started scap: SWAT: Clarify and expand messages mentioning loss of session data [[gerrit:269424]] |
[production] |
| 16:18 |
<thcipriani@mira> |
Synchronized wmf-config: SWAT: Enable ArticlePlaceholder on test wikis [[gerrit:269399]] (duration: 01m 19s) |
[production] |
| 16:15 |
<thcipriani> |
mw1037.eqiad.wmnet error during SWAT rsync: failed to set times on "/srv/mediawiki/.": Read-only file system (30) |
[production] |
| 16:09 |
<thcipriani@mira> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable math data type on Wikidata and everywhere [[gerrit:269398]] (duration: 02m 31s) |
[production] |
| 16:06 |
<bd808> |
Deleted corrupt integration-slave-precise-1003:/mnt/jenkins-workspace/workspace/mediawiki-core-php53lint/.git |
[releng] |
| 15:59 |
<elukey> |
puppet re-enabled on kafka1012 |
[production] |
| 15:56 |
<paravoid> |
"power"cycling alsafi |
[production] |
| 15:55 |
<moritzm> |
uploaded linux 4.4-1~wmf1 (jessie-wikimedia/experimental) to carbon |
[production] |
| 15:53 |
<elukey> |
restarted kafka1012 with 48hrs of log retention |
[analytics] |
| 15:47 |
<_joe_> |
re-removed the puppet facts for protactinium |
[production] |
| 15:41 |
<elukey> |
kafka broker restarted - kafka1012 |
[analytics] |
| 15:40 |
<paravoid> |
echo 1 > /proc/sys/net/ipv4/vs/schedule_icmp on lvs3001 |
[production] |
| 15:36 |
<elukey> |
disabled puppet on kafka1012, changing temporary kafka retention to purge some extra logs |
[production] |
| 15:17 |
<cmjohnson1> |
snapshot1002 mistakenly taken offline -- booting now |
[production] |
| 15:15 |
<paravoid> |
upgrading lvs4001/4002 to linux 4.4.0 |
[production] |
| 15:11 |
<hashar> |
mira: /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.13 php-1.27.0-wmf.13 |
[releng] |
| 15:07 |
<godog> |
stop cassandra on restbase1007, cpu/mem upgrade and reimage |
[production] |
| 14:59 |
<paravoid> |
upgrading lvs3001/3002 to linux 4.4.0 |
[production] |
| 14:53 |
<godog> |
reboot ms-be1004, xfs hosed |
[production] |
| 14:51 |
<hashar> |
Cutting branches 1.27.0-wmf.13 |
[production] |
| 14:51 |
<hashar> |
./make-wmf-branch -n 1.27.0-wmf.13 -o master |
[releng] |
| 14:50 |
<hashar> |
pooling back integration-slave-precise1001 - 1004. Manually fetched git repos in workspace for mediawiki core php53 |
[releng] |
| 14:49 |
<hashar> |
make-wmf-branch instance: created a local ssh key pair and set the config to use User: hashar |
[releng] |
| 14:46 |
<elukey> |
re-enabled puppet on mc1004.eqiad |
[production] |
| 14:45 |
<bblack> |
resuming cpNNNN rolling kernel reboots |
[production] |
| 14:41 |
<_joe_> |
setting mw1026-1050 as inactive in the appservers pool (T126242) |
[production] |
| 14:13 |
<hashar> |
pooling https://integration.wikimedia.org/ci/computer/integration-slave-precise-1012/ Mysql is back .. Blame puppet |
[releng] |
| 14:12 |
<hashar> |
de pooling https://integration.wikimedia.org/ci/computer/integration-slave-precise-1012/ Mysql is gone somehow |
[releng] |
| 14:04 |
<hashar> |
Manually git fetching mediawiki-core in /mnt/jenkins-workspace/workspace/mediawiki-core-php53lint of slaves precise 1001 to 1004 (git on Precise is remarkably too slow) |
[releng] |
| 13:58 |
<hashar> |
shutting down jenkins finally, and restarting it |
[production] |
| 13:51 |
<hashar> |
Restarting Jenkins. It can not manage to add slaves |
[production] |
| 13:28 |
<hashar> |
salt '*trusty*' cmd.run 'update-alternatives --set php /usr/bin/hhvm' |
[releng] |
| 13:28 |
<hashar> |
salt '*precise*' cmd.run 'update-alternatives --set php /usr/bin/php5' |
[releng] |
| 13:17 |
<hashar> |
salt -v --batch=3 '*slave*' cmd.run 'puppet agent -tv' |
[releng] |
| 13:15 |
<paravoid> |
upgrading lvs1001/lvs1007/lvs1002/lvs1008/lvs1003/lvs1009 to 4.4.0 |
[production] |
| 13:15 |
<hashar> |
removing https://gerrit.wikimedia.org/r/#/c/269370/ from CI puppet master |
[releng] |
| 13:14 |
<hashar> |
slave recurse infinitely doing /bin/bash -eu /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh then loop over /bin/bash /usr/bin/php maintenance/install.php --confpath /mnt/jenkins-workspace/workspace/mediawiki-core-qunit/src --dbtype=mysql --dbserver=127.0.0.1:3306 --dbuser=jenkins_u2 --dbpass=pw_jenkins_u2 --dbname=jenkins_u2_mw --pass testpass TestWiki WikiAdmin https://phabricator.wikimedia.org/T126327 |
[releng] |
| 13:11 |
<akosiaris> |
reboot serpens to apply memory increase of 2G |
[production] |
| 13:07 |
<paravoid> |
installing linux 4.4.0 on lvs1001 |
[production] |
| 13:01 |
<hashar> |
Jenkins disabled again :( |
[production] |
| 12:53 |
<akosiaris> |
reboot seaborgium to apply memory increase of 2G |
[production] |
| 12:47 |
<hashar> |
Updated faulty script that caused 'php' too loop infinitely. Jenkins back up. |
[production] |
| 12:46 |
<hashar> |
Mass testing php loop of death: salt -v '*slave*' cmd.run 'timeout 2s /srv/deployment/integration/slave-scripts/bin/php --version' |
[releng] |
| 12:40 |
<hashar> |
mass rebooting CI slaves from wikitech |
[releng] |
| 12:39 |
<hashar> |
salt -v '*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" |
[releng] |
| 12:36 |
<hashar> |
Jenkins no more accept new jobs until the slaves are fixed :/ |
[production] |
| 12:33 |
<hashar> |
all CI slaves looping to death because of a php loop |
[production] |
| 12:33 |
<hashar> |
all slaves dieing due to PHP looping |
[releng] |