2016-03-09
§
|
12:25 |
<elukey> |
added rdb1003 back to the Job Runners queue. All the jobchron processes on jobrunners/videoscalers need to be restarted. |
[production] |
12:07 |
<elukey@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: Add rdb1003 back to the Redis JobQueue pool after maintenance (duration: 00m 34s) |
[production] |
12:00 |
<elukey> |
re-enabled puppet on rdb1004 |
[production] |
11:47 |
<moritzm> |
uploaded firmware-nonfree 20151018 to jessie-wikimedia on carbon |
[production] |
11:39 |
<godog> |
rolling restart cassandra in staging after merging https://gerrit.wikimedia.org/r/#/c/275917/ |
[production] |
11:27 |
<jynus> |
setting up master-master cross-datacenter replication for s2 (db1018-db2017) |
[production] |
11:20 |
<ema> |
hhvm restarted on mw1140 |
[production] |
11:04 |
<jynus> |
deployed manually new code at dbtree (noc.wikimedia.org) |
[production] |
10:39 |
<elukey> |
puppet disabled on rdb1004 (plus Redis instances) as precautionary step for master reimage - rdb1003 |
[production] |
10:35 |
<jynus> |
setting up master-master replication on tools (labsdb100[45]) |
[production] |
09:32 |
<moritzm> |
rebooting iron for kernel update |
[production] |
09:26 |
<elukey> |
puppet disabled on rdb1003 for debian re-image. Redis servers going to be stopped too as pre-step for backup. |
[production] |
08:32 |
<oblivian@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: re-routing writes from rdb1003 to rdb1005 for reimaging (duration: 00m 36s) |
[production] |
08:29 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2035 (duration: 00m 39s) |
[production] |
07:49 |
<_joe_> |
uploading hhvm 3.12, extensions to reprepro |
[production] |
06:26 |
<mutante> |
bast2001 - still installing in snail mode - please feel free to check if it's done, and if so re-add to puppet so users get created.thx |
[production] |
03:16 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Mar 9 03:16:21 UTC 2016 (duration 8m 2s) |
[production] |
03:08 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.16) (duration: 17m 51s) |
[production] |
02:32 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.15) (duration: 13m 28s) |
[production] |
02:13 |
<ori@tin> |
Synchronized php-1.27.0-wmf.15/includes/diff/DairikiDiff.php: I4d4b8f81c: Dont quote assert expressions in DairikiDiff (duration: 00m 27s) |
[production] |
02:12 |
<ori@tin> |
Synchronized php-1.27.0-wmf.16/includes/diff/DairikiDiff.php: I4d4b8f81c: Dont quote assert expressions in DairikiDiff (duration: 00m 31s) |
[production] |
01:40 |
<csteipp> |
redeploy patch for T129120 |
[production] |
01:39 |
<csteipp@tin> |
Synchronized php-1.27.0-wmf.16/includes/api/ApiQueryInfo.php: (no message) (duration: 00m 33s) |
[production] |
00:33 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.15/extensions/Echo/: SWAT (duration: 00m 32s) |
[production] |
00:32 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.15/extensions/MobileFrontend/: SWAT (duration: 00m 32s) |
[production] |
00:30 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.16/extensions/MobileFrontend/: SWAT (duration: 00m 29s) |
[production] |
00:29 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.16/extensions/Echo/: SWAT (duration: 00m 38s) |
[production] |
00:25 |
<maxsem@tin> |
Synchronized wmf-config/: https://gerrit.wikimedia.org/r/275378 - comment only (duration: 00m 27s) |
[production] |
00:16 |
<maxsem@tin> |
Synchronized wmf-config/filebackend-production.php: https://gerrit.wikimedia.org/r/272922 (duration: 00m 27s) |
[production] |
00:13 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/275858/ (duration: 00m 28s) |
[production] |
00:09 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/275275 (duration: 00m 29s) |
[production] |
2016-03-08
§
|
23:20 |
<mutante> |
bast2001 - install issues - extending downtime, bbiaw |
[production] |
23:20 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.16 |
[production] |
23:11 |
<mutante> |
neon: re-enable puppet, start icinga-wm |
[production] |
23:03 |
<twentyafterfour@tin> |
Synchronized w/static/: (no message) (duration: 00m 32s) |
[production] |
22:53 |
<mutante> |
bast2001 - powercycle, reinstall |
[production] |
22:19 |
<twentyafterfour@tin> |
Finished scap: testwiki to php-1.27.0-wmf.16 and rebuild l10n cache (duration: 26m 06s) |
[production] |
22:04 |
<ori> |
Canary application servers (mw1017-mw1025) and canary API application servers (mw1114-mw1119) upgraded to HHVM 3.12.1 |
[production] |
22:04 |
<jynus> |
recreating slave watchdog events on all servers |
[production] |
21:53 |
<twentyafterfour@tin> |
Started scap: testwiki to php-1.27.0-wmf.16 and rebuild l10n cache |
[production] |
21:02 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 34s) |
[production] |
21:01 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1066 (duration: 00m 33s) |
[production] |
20:49 |
<ori> |
Updated mw1200 to HHVM 3.12.1; repooling |
[production] |
20:45 |
<ori> |
Depooled mw1200 for HHVM update |
[production] |
20:40 |
<jynus> |
killing long running queries on db1066 |
[production] |
20:39 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1066 (duration: 01m 49s) |
[production] |
18:55 |
<mutante> |
temp. stopped icinga-wm |
[production] |
18:54 |
<moritzm> |
uploaded linux 4.4.2-3+wmf1/jessie-wikimedia (based on Linux 4.4.4) to carbon |
[production] |
17:46 |
<bblack> |
switching text cache routing: ulsfo->codfw |
[production] |
17:36 |
<papaul> |
sinistra signing puppet certs, salt-key, initial run |
[production] |