|
2018-04-03
ยง
|
| 23:55 |
<XioNoX> |
re-activating graceful-switchover on cr1-codfw - T189588 |
[production] |
| 23:16 |
<dereckson@tin> |
Synchronized wmf-config/CommonSettings.php: Make a note about the loading order of GlobalPreferences and Echo ([[Gerrit:422642]]) (no-op) (duration: 01m 17s) |
[production] |
| 23:10 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Rollout VirtualPageViews (final stage) (T189906) (duration: 01m 19s) |
[production] |
| 22:34 |
<mutante> |
cobalt - puppet disabled temporarily to apply fix to "simplify directory structure" change .. on gerrit2001 first |
[production] |
| 22:25 |
<mutante> |
restarting Apache on phab1001 - T182832 |
[production] |
| 22:14 |
<twentyafterfour> |
Finished MediaWiki Train for group0, 1.31.0-wmf.28 refs T183967 |
[production] |
| 22:12 |
<twentyafterfour@tin> |
Pruned MediaWiki: 1.31.0-wmf.25 [keeping static files] (duration: 01m 55s) |
[production] |
| 22:10 |
<twentyafterfour@tin> |
Pruned MediaWiki: 1.31.0-wmf.24 [keeping static files] (duration: 04m 18s) |
[production] |
| 21:30 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.31.0-wmf.28 refs T183967 |
[production] |
| 21:15 |
<twentyafterfour@tin> |
Finished scap: testwikis wikis to 1.31.0-wmf.28 refs T183967 (duration: 46m 38s) |
[production] |
| 21:13 |
<urandom> |
(re)starting restbase-dev1004-{a,b} (ooms), and enabling alternately patched cassandra 3.11.2 build - T186751 |
[production] |
| 20:28 |
<twentyafterfour@tin> |
Started scap: testwikis wikis to 1.31.0-wmf.28 refs T183967 |
[production] |
| 20:22 |
<ejegg> |
disabled thank you mail sender |
[production] |
| 20:11 |
<ottomata> |
bouncing main -> jumbo mirrormaker to apply batch.size = 65536 |
[analytics] |
| 19:46 |
<urandom> |
restarting restbase-dev1004-{a,b} to enable patched cassandra 3.11.2 build - T186751 |
[production] |
| 19:32 |
<ottomata> |
bouncing main -> jumbo MirrorMaker unsetting http://session.timeout.ms/, this has a restiction on the broker in 0.9 :( |
[analytics] |
| 19:22 |
<ottomata> |
bouncing main -> jumbo MirrorMaker setting session.timeout.ms = 125000 |
[analytics] |
| 19:07 |
<twentyafterfour> |
Preparing to deploy 1.31.0-wmf.28 refs T183967 |
[production] |
| 18:46 |
<ottomata> |
restart main -> jumbo MirrorMaker with request.timeout.ms = 2 minutes |
[analytics] |
| 18:25 |
<urandom> |
upgrading restbase-dev1006-b to cassandra 3.11.2 - T186751 |
[production] |
| 18:23 |
<urandom> |
upgrading restbase-dev1006-a to cassandra 3.11.2 - T186751 |
[production] |
| 18:20 |
<urandom> |
upgrading restbase-dev1005-b to cassandra 3.11.2 - T186751 |
[production] |
| 18:18 |
<urandom> |
upgrading restbase-dev1005-a to cassandra 3.11.2 - T186751 |
[production] |
| 18:15 |
<urandom> |
upgrading restbase-dev1004-b to cassandra 3.11.2 - T186751 |
[production] |
| 18:13 |
<urandom> |
upgrading restbase-dev1004-a to cassandra 3.11.2 - T186751 |
[production] |
| 18:05 |
<mutante> |
rhodium - closing idle screen session from maintenance work on puppetmasters |
[production] |
| 18:03 |
<mutante> |
elnath - fixing and re-enabling Icinga alert about screens, none are running, spare hosts should not have these |
[production] |
| 17:59 |
<mutante> |
restarting ferm on bromine |
[production] |
| 17:40 |
<elukey> |
manually set net.ipv4.tcp_tw_reuse=1 on restbase1007 as test for T190213 |
[production] |
| 17:35 |
<sbisson@tin> |
Finished deploy [tilerator/deploy@8e68cb8]: Deploying tilerator i18n to maps-test* (take 2) (duration: 00m 25s) |
[production] |
| 17:35 |
<sbisson@tin> |
Started deploy [tilerator/deploy@8e68cb8]: Deploying tilerator i18n to maps-test* (take 2) |
[production] |
| 17:28 |
<awight@tin> |
Finished deploy [ores/deploy@7701cee]: ORES versioned virtualenv, T181071 (duration: 01m 27s) |
[production] |
| 17:28 |
<sbisson@tin> |
Finished deploy [tilerator/deploy@03add2d]: Deploying tilerator i18n to maps-test* (duration: 04m 09s) |
[production] |
| 17:27 |
<awight@tin> |
Started deploy [ores/deploy@7701cee]: ORES versioned virtualenv, T181071 |
[production] |
| 17:25 |
<awight@tin> |
Finished deploy [ores/deploy@7701cee]: ORES versioned virtualenv, T181071 (duration: 00m 27s) |
[production] |
| 17:25 |
<awight@tin> |
Started deploy [ores/deploy@7701cee]: ORES versioned virtualenv, T181071 |
[production] |
| 17:24 |
<sbisson@tin> |
Started deploy [tilerator/deploy@03add2d]: Deploying tilerator i18n to maps-test* |
[production] |
| 17:24 |
<moritzm> |
upgrading HHVM on labweb* |
[production] |
| 17:18 |
<jynus> |
reloading labsdb proxy configuration |
[production] |
| 17:08 |
<elukey> |
manually set net.ipv4.tcp_tw_reuse=1 on restbase2001 as test for T190213 |
[production] |
| 16:53 |
<demon@tin> |
Finished deploy [gerrit/gerrit@aa1a1a0]: no-op, pushing empty motd.config file (duration: 00m 11s) |
[production] |
| 16:53 |
<demon@tin> |
Started deploy [gerrit/gerrit@aa1a1a0]: no-op, pushing empty motd.config file |
[production] |
| 16:33 |
<urandom> |
rebooting restbase-dev1006 - T186751 |
[production] |
| 16:10 |
<urandom> |
rebooting restbase-dev1004 - T186751 |
[production] |
| 16:04 |
<ariel@tin> |
Finished deploy [dumps/dumps@77dc467]: split up some large modules, prep work for prefetch changes (duration: 00m 04s) |
[production] |
| 16:04 |
<ariel@tin> |
Started deploy [dumps/dumps@77dc467]: split up some large modules, prep work for prefetch changes |
[production] |
| 15:39 |
<elukey> |
roll restart of zookeeper on conf100[123] to pick up prometheus monitoring |
[production] |
| 15:26 |
<elukey> |
manually run hdfs balancer on an1003 (tmux session) |
[analytics] |
| 15:25 |
<elukey> |
killed a jvm belonging to hdfs-balancer stuck from march 9th |
[analytics] |
| 15:09 |
<godog> |
depool ms-fe2005 to test rewrite.py - T183902 |
[production] |