|
2016-04-19
§
|
| 13:55 |
<ori> |
[switchover #1]: disabling eqiad jobrunners via "salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'service jobrunner stop; service jobchron stop;'". |
[production] |
| 13:47 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool one db per shard as a backup (duration: 00m 27s) |
[production] |
| 11:33 |
<volans> |
changing binlog_format to STATEMENT for codfw masters for shards s1-s7 T124699 |
[production] |
| 10:07 |
<jynus> |
updated dns entries about mysql masters |
[production] |
| 09:16 |
<godog> |
shutdown restbase100[56] |
[production] |
| 09:11 |
<godog> |
stop cassandra and restbase on restbase1006 |
[production] |
| 09:06 |
<godog> |
stop compactions on restbase1014 |
[production] |
| 08:33 |
<joal> |
deployed new refinery on hadoop |
[analytics] |
| 08:21 |
<joal> |
deploying refinery from tin |
[analytics] |
| 03:13 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 28s) |
[production] |
| 03:12 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2006 (duration: 00m 31s) |
[production] |
| 02:54 |
<ori@tin> |
Synchronized php-1.27.0-wmf.21/includes/api/ApiStashEdit.php: Ie9799f5ea: Segment stash edit cache stats by basis for hit/miss (duration: 00m 39s) |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Apr 19 02:31:46 UTC 2016 (duration 9m 46s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.21) (duration: 09m 47s) |
[production] |
| 02:06 |
<urandom> |
systemctl mask cassandra-b on restbase2004.codfw.wmnet (it should not be running) |
[production] |
| 01:34 |
<mutante> |
restbase2004 - starting crashed cassandra-b service |
[production] |
| 01:11 |
<mutante> |
restbase2004 - unit cassandra-b is failed |
[production] |
| 00:42 |
<mutante> |
kraz - signing puppet certs, adding salt keys |
[production] |
| 00:29 |
<mutante> |
kraz.codfw.wmnet - initial install, adding to site |
[production] |
|
2016-04-18
§
|
| 23:33 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: I1547834: Remove unused ['bits'] configuration (duration: 00m 27s) |
[production] |
| 23:22 |
<YuviPanda> |
running sync-common on mw1145 to catch up on deploys |
[production] |
| 23:17 |
<YuviPanda> |
rebooted mw1145 from mgmt |
[production] |
| 23:14 |
<ori@tin> |
Synchronized wmf-config: I0ec3c015f: Update parser cache configuration for tag-based hashing (duration: 01m 48s) |
[production] |
| 23:08 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2006 (duration: 01m 01s) |
[production] |
| 23:06 |
<ori@tin> |
Synchronized php-1.27.0-wmf.21/includes/objectcache/SqlBagOStuff.php: Ie9799f5ea: Allow tag names for SqlBagOStuff consistent hashing (duration: 04m 08s) |
[production] |
| 22:52 |
<krinkle@tin> |
Synchronized docroot/noc/conf: noc: remote bits references (duration: 00m 32s) |
[production] |
| 22:45 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2005 (duration: 00m 33s) |
[production] |
| 22:44 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 26s) |
[production] |
| 22:20 |
<chasemp> |
resetting labcontrol1001 puppet master with auth.conf which is fixing all the puppet clients in Labs |
[production] |
| 21:33 |
<gehel> |
deploying latest wdqs |
[production] |
| 21:29 |
<papaul> |
conf200[1-3] - signing puppet certs, salt-key, initial run |
[production] |
| 20:58 |
<papaul> |
OS install on conf200[1-3] |
[production] |
| 20:55 |
<mutante> |
furud.codfw - initial puppet install, signing certs |
[production] |
| 18:53 |
<bd808> |
Ran ~bd808/cleanup-var-crap.sh to free a bit of space on deployment-mediawiki01; this instance really needs a bigger disk |
[releng] |
| 18:47 |
<bd808> |
deployment-mediawiki01: rm /var/log/apache2/*.gz |
[releng] |
| 18:43 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 28s) |
[production] |
| 18:42 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 26s) |
[production] |
| 18:37 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2004 (duration: 00m 25s) |
[production] |
| 18:27 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1004 (duration: 00m 36s) |
[production] |
| 17:31 |
<elukey> |
restarted kafkatee on oxygen after module upgrade (was stuck from this morning due to a kafka broker restart) |
[production] |
| 15:31 |
<andrewbogott> |
migration integration-slave-trusty-1018 to labvirt1009 |
[integration] |
| 14:51 |
<jzerebecki> |
reloading zuul for d9e80bd..113ea21 |
[releng] |
| 14:21 |
<ottomata> |
upgraded pykafka on hafnium and restarted statsv |
[production] |
| 14:04 |
<andrewbogott> |
puppet agent -tv on labvirt1002 which restarted nrpe and resolved icinga alerts |
[production] |
| 14:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool pc1004 (duration: 00m 29s) |
[production] |
| 14:01 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2004 (duration: 00m 27s) |
[production] |
| 12:59 |
<jzerebecki> |
integration-slave-trusty-1014:~$ sudo -i rm -f /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/GlobalCssJs/.git/refs/heads/wmf/1.27.0-wmf.11.lock |
[releng] |
| 12:57 |
<jzerebecki> |
integration-slave-trusty-1014:~$ sudo -i 'rm -f /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/GlobalCssJs/.git/refs/heads/wmf/1.27.0-wmf.11.lock' |
[releng] |
| 11:30 |
<mobrovac> |
restbase restarting eqiad after https://gerrit.wikimedia.org/r/#/c/283958/ |
[production] |
| 10:11 |
<elukey> |
execute sudo eventloggingctl restart on eventlogging1001 |
[analytics] |