|
2016-04-19
§
|
| 14:10 |
<_joe_> |
[switchover #6] wiped memcached |
[production] |
| 14:10 |
<jynus> |
[switchover #5] DB Masters on eqiad set as read-only, and confirmed it |
[production] |
| 14:09 |
<_joe_> |
[switchover #6] disabled puppet on all redis hosts as a safety measure before inverting replication after the puppet change |
[production] |
| 14:03 |
<paravoid> |
sites in planned readonly-mode, cf. http://blog.wikimedia.org/2016/04/18/wikimedia-server-switch/ |
[production] |
| 14:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw (duration: 00m 35s) |
[production] |
| 14:01 |
<jynus> |
[switchover #4] Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw |
[production] |
| 13:56 |
<_joe_> |
[switchover #3] disabling cronjobs on terbium |
[production] |
| 13:55 |
<ori> |
[switchover #1]: disabling eqiad jobrunners via "salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'service jobrunner stop; service jobchron stop;'". |
[production] |
| 13:47 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool one db per shard as a backup (duration: 00m 27s) |
[production] |
| 11:33 |
<volans> |
changing binlog_format to STATEMENT for codfw masters for shards s1-s7 T124699 |
[production] |
| 10:07 |
<jynus> |
updated dns entries about mysql masters |
[production] |
| 09:16 |
<godog> |
shutdown restbase100[56] |
[production] |
| 09:11 |
<godog> |
stop cassandra and restbase on restbase1006 |
[production] |
| 09:06 |
<godog> |
stop compactions on restbase1014 |
[production] |
| 03:13 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 28s) |
[production] |
| 03:12 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2006 (duration: 00m 31s) |
[production] |
| 02:54 |
<ori@tin> |
Synchronized php-1.27.0-wmf.21/includes/api/ApiStashEdit.php: Ie9799f5ea: Segment stash edit cache stats by basis for hit/miss (duration: 00m 39s) |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Apr 19 02:31:46 UTC 2016 (duration 9m 46s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.21) (duration: 09m 47s) |
[production] |
| 02:06 |
<urandom> |
systemctl mask cassandra-b on restbase2004.codfw.wmnet (it should not be running) |
[production] |
| 01:34 |
<mutante> |
restbase2004 - starting crashed cassandra-b service |
[production] |
| 01:11 |
<mutante> |
restbase2004 - unit cassandra-b is failed |
[production] |
| 00:42 |
<mutante> |
kraz - signing puppet certs, adding salt keys |
[production] |
| 00:29 |
<mutante> |
kraz.codfw.wmnet - initial install, adding to site |
[production] |
|
2016-04-18
§
|
| 23:33 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: I1547834: Remove unused ['bits'] configuration (duration: 00m 27s) |
[production] |
| 23:22 |
<YuviPanda> |
running sync-common on mw1145 to catch up on deploys |
[production] |
| 23:17 |
<YuviPanda> |
rebooted mw1145 from mgmt |
[production] |
| 23:14 |
<ori@tin> |
Synchronized wmf-config: I0ec3c015f: Update parser cache configuration for tag-based hashing (duration: 01m 48s) |
[production] |
| 23:08 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2006 (duration: 01m 01s) |
[production] |
| 23:06 |
<ori@tin> |
Synchronized php-1.27.0-wmf.21/includes/objectcache/SqlBagOStuff.php: Ie9799f5ea: Allow tag names for SqlBagOStuff consistent hashing (duration: 04m 08s) |
[production] |
| 22:52 |
<krinkle@tin> |
Synchronized docroot/noc/conf: noc: remote bits references (duration: 00m 32s) |
[production] |
| 22:45 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2005 (duration: 00m 33s) |
[production] |
| 22:44 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 26s) |
[production] |
| 22:20 |
<chasemp> |
resetting labcontrol1001 puppet master with auth.conf which is fixing all the puppet clients in Labs |
[production] |
| 21:33 |
<gehel> |
deploying latest wdqs |
[production] |
| 21:29 |
<papaul> |
conf200[1-3] - signing puppet certs, salt-key, initial run |
[production] |
| 20:58 |
<papaul> |
OS install on conf200[1-3] |
[production] |
| 20:55 |
<mutante> |
furud.codfw - initial puppet install, signing certs |
[production] |
| 18:43 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 28s) |
[production] |
| 18:42 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 26s) |
[production] |
| 18:37 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2004 (duration: 00m 25s) |
[production] |
| 18:27 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1004 (duration: 00m 36s) |
[production] |
| 17:31 |
<elukey> |
restarted kafkatee on oxygen after module upgrade (was stuck from this morning due to a kafka broker restart) |
[production] |
| 14:21 |
<ottomata> |
upgraded pykafka on hafnium and restarted statsv |
[production] |
| 14:04 |
<andrewbogott> |
puppet agent -tv on labvirt1002 which restarted nrpe and resolved icinga alerts |
[production] |
| 14:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool pc1004 (duration: 00m 29s) |
[production] |
| 14:01 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2004 (duration: 00m 27s) |
[production] |
| 11:30 |
<mobrovac> |
restbase restarting eqiad after https://gerrit.wikimedia.org/r/#/c/283958/ |
[production] |
| 10:07 |
<_joe_> |
traffic from eqiad caches switched to codfw for restbase,citoid,cxserver |
[production] |
| 09:43 |
<moritzm> |
installing openssh updates on jessie systems |
[production] |