351-400 of 10000 results (49ms)
2018-05-18 §
15:38 <reedy@tin> Synchronized php-1.32.0-wmf.3/extensions/VisualEditor/: Fix dialog (duration: 01m 25s) [production]
15:31 <gehel> rolling restart of cassandra on maps1* (repair was started on each node, instead of sequentially) [production]
15:23 <gehel> clear cassandra snapshots on maps1002 [production]
13:59 <marostegui> Manually fail disk #6 on db1066 - T194870 [production]
13:19 <bawolff_> reset 2FA for Trizek_(WMF) [production]
12:26 <jynus> stop and reimage db2041 [production]
10:15 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1085 with low load (duration: 01m 20s) [production]
09:47 <marostegui> Stop MySQL on db1116 for testing [production]
09:44 <marostegui> Stop MySQL on db2092 for testing [production]
09:43 <hoo> Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds [production]
09:33 <gehel> cleared v3 snapshot on maps servers [production]
09:25 <jynus> stop and reimage db1085 [production]
09:24 <gehel> drop v3 keyspace on cassandra maps (unused since migration to i18n) [production]
09:01 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 01m 20s) [production]
08:32 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1105:s6 and other vslow hosts (duration: 01m 21s) [production]
06:14 <XioNoX> bumping eqsin-codfw link OSPF metric to 5000 (due to packet loss on link) [production]
05:37 <marostegui> Stop MySQL on db1120 to copy its content to db2075 - T190704 [production]
05:33 <marostegui> Deploy schema change on dbstore1002:s3 - T191519 T188299 T190148 [production]
05:25 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1066 - T193847 (duration: 01m 22s) [production]
05:24 <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1011 [production]
05:18 <marostegui> Stop MySQL and reboot db1067 - T194852 [production]
01:34 <twentyafterfour@tin> Synchronized php-1.32.0-wmf.4: sync wmf.4 to deploy https://gerrit.wikimedia.org/r/#/c/433673/ (duration: 09m 54s) [production]
01:27 <twentyafterfour> syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050 [production]
00:19 <mutante> rdb2004 - down in Icinga since >1d, nothing on console, dont see a SAL entry. powercycling [production]
2018-05-17 §
23:35 <twentyafterfour> MediaWiki Train for 1.32.0-wmf.4 remains blocked by critical bugs, see T191050 for a list of blockers. [production]
23:34 <twentyafterfour@tin> Synchronized php: group1 wikis to 1.32.0-wmf.3 refs T191050 (duration: 01m 20s) [production]
23:32 <twentyafterfour@tin> rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3 refs T191050 [production]
23:29 <twentyafterfour> rolling back [production]
23:29 <twentyafterfour> still seeing Notice: Undefined variable: nonce in /srv/mediawiki/php-1.32.0-wmf.4/includes/resourceloader/ResourceLoaderClientHtml.php on line 272 [production]
23:28 <twentyafterfour@tin> Synchronized php: group1 wikis to 1.32.0-wmf.4 refs T191050 (duration: 01m 17s) [production]
23:26 <twentyafterfour@tin> rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4 refs T191050 [production]
23:22 <twentyafterfour@tin> Synchronized php-1.32.0-wmf.4/: sync https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 (duration: 09m 54s) [production]
22:53 <twentyafterfour> deploying https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050 [production]
19:46 <twentyafterfour@tin> Synchronized php: group1 wikis to 1.32.0-wmf.3 (duration: 01m 20s) [production]
19:44 <twentyafterfour@tin> rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3 [production]
19:41 <twentyafterfour> rolling back due to spike of undefined variable notices in resourceloader and ApiCSPReport.php [production]
19:39 <twentyafterfour@tin> Synchronized php: group1 wikis to 1.32.0-wmf.4 (duration: 01m 21s) [production]
19:38 <twentyafterfour@tin> rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4 [production]
19:33 <twentyafterfour> getting the train back on track. Starting with group1 to 1.32.0-wmf.4 right now, will do all wikis to wmf.4 after verifying that group1 looks stable. [production]
19:28 <twentyafterfour@tin> Synchronized php-1.32.0-wmf.4/extensions/Echo/: unbreak T194848 (duration: 01m 24s) [production]
19:11 <twentyafterfour> train is still blocked by T194848 [production]
17:23 <arlolra> Updated Parsoid to fd49ab4 (T194821, T194687) [production]
17:15 <arlolra@tin> Finished deploy [parsoid/deploy@091b891]: Updating Parsoid to fd49ab4 (duration: 09m 35s) [production]
17:06 <arlolra@tin> Started deploy [parsoid/deploy@091b891]: Updating Parsoid to fd49ab4 [production]
16:11 <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1011 T174047 T194341 [production]
16:04 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 01m 21s) [production]
15:29 <marostegui> Manually fail disk #6 on db1064 to get it replaced [production]
15:28 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1093 with full weight (duration: 01m 21s) [production]
15:00 <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1010 [production]
14:39 <papaul> shutting down furud for shelves swap [production]