2801-2850 of 10000 results (32ms)
2016-06-16 §
00:57 <twentyafterfour> puppet disabled on iridium because https://gerrit.wikimedia.org/r/#/c/294653/ needs to merge (hotfix in preamble.php which puppet will undo if it's allowed to run) [production]
00:43 <twentyafterfour> phabricator upgrade/maintenance complete. Everything appears to be back up and running normally. [production]
00:41 <twentyafterfour> taking phabricator offline momentarily for scheduled maintenance. [production]
00:24 <robh> mw1147 rebooted and manually running scap pull [production]
00:21 <robh> mw1147 seems to have died during scap, unresponsive from serial console, powercycled [production]
00:16 <mattflaschen@tin> Synchronized php-1.28.0-wmf.6/extensions/Kartographer: Search for maplinks inside and outside of content. (duration: 01m 08s) [production]
2016-06-15 §
23:37 <mattflaschen@tin> Synchronized php-1.28.0-wmf.6/extensions/Echo: Sync Echo fix for cross-wiki notifications: 62324e3 (duration: 00m 33s) [production]
21:32 <aaron@tin> Synchronized wmf-config/filebackend-production.php: Set "sync" filebackend replication to measure latency effect (duration: 00m 25s) [production]
21:27 <aaron@tin> Synchronized php-1.28.0-wmf.6/includes/libs/objectcache/WANObjectCache.php: faff8f1ef1bfefd1804a3f46e58566711faa3224 (duration: 00m 27s) [production]
21:15 <dapatrick> Deployed patch for T137264 to wmf.5 and wmf.6 [production]
20:17 <hashar@tin> Synchronized wmf-config/throttle.php: Temporary IP Cap Lift on es.wiki T137917 (duration: 00m 30s) [production]
20:09 <subbu> finished deploying parsoid sha 3445eceb [production]
20:05 <bblack> cache frontend restarts complete [production]
20:04 <subbu> synced new code; restarted parsoid on wtp1001 as a canary [production]
20:02 <subbu> starting parsoid deploy [production]
19:25 <bblack> rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~35 mins to completion - T107236 (...._ [production]
19:15 <bblack> varnish frontend restart halted - v4 compat issue to address :P [production]
19:11 <bblack> rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~30 mins to completion - T107236 [production]
19:05 <hashar@tin> rebuilt wikiversions.php and synchronized wikiversions files: (no message) [production]
18:54 <ori> Started MySQL on es2019 (T130702) [production]
16:32 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1023; pool db1085 (disabled), db1088, db1092 w/low weight (duration: 00m 25s) [production]
16:07 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294514|Fix autopatrolled group for ko.wikipedia]] (duration: 00m 31s) [production]
16:00 <thcipriani@tin> Synchronized php-1.28.0-wmf.6/resources/src/mediawiki.special/mediawiki.special.search.styles.css: SWAT: [[gerrit:294511|Explicitly specify the width of the search input on Special:Search]] (duration: 00m 25s) [production]
15:52 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294254|Add autopatrolled group in kowiki]] (duration: 00m 24s) [production]
15:33 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294488|Deploy ORES beta feature in wikidatawiki]] (duration: 00m 24s) [production]
15:23 <elukey@palladium> conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet [production]
15:23 <thcipriani@tin> Synchronized php-1.28.0-wmf.5/extensions/ORES: SWAT: [[gerrit:294491|Skip when an edit is errored in PopulateDatabase.php]] (duration: 00m 27s) [production]
15:17 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294462|Send authentication events to logstash]] (duration: 00m 28s) [production]
15:15 <elukey@palladium> conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet [production]
15:11 <thcipriani@tin> Synchronized wmf-config/logging.php: SWAT: [[gerrit:293701|Fix logging config for authmanager metrics channel rename]] (duration: 00m 24s) [production]
15:10 <elukey@palladium> conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet [production]
15:06 <thcipriani@tin> Synchronized wmf-config/throttle.php: SWAT: [[gerrit:293970|Remove old throttle rules]] (duration: 00m 30s) [production]
15:00 <elukey@palladium> conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet [production]
15:00 <mobrovac> scb disabled puppet for stopped change-prop during kafka nodes upgrade [production]
15:00 <elukey> rebooting Eqiad Event Bus for kernel upgrades (one node at the time) [production]
14:24 <moritzm> installing php security updates on jessie systems [production]
13:55 <moritzm> remove unused PHP packages from the recently provisioned jessie app servers (new installation are fixed in puppet to only install php5-cli, but the initial set needs fixed up manually) [production]
13:40 <gehel> rolling back update of firejail on maps2001 [production]
13:16 <_joe_> stopped jobchron, jobrunner on mw1299, masked in systemd [production]
13:15 <mobrovac> change-prop deployed 6ad337 [production]
13:06 <moritzm> installing libav security updates [production]
12:37 <_joe_> rebooting mw1299 [production]
12:06 <gehel> upgrade of firejail on maps server stopped, pending a patch to service::node [production]
11:46 <mobrovac> scb enabled puppet back [production]
11:44 <gehel> upgrading firejail to 0.9.38 on maps servers [production]
11:32 <mobrovac> scb disabled puppet for 5 min to keep change-prop down [production]
11:30 <mobrovac> change-prop deploying 353b926 [production]
11:29 <jynus> stopping db1023 for cloning to new s6 hosts [production]
11:22 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Increase new enwiki dbs weight, depool db1023 for cloning (duration: 00m 27s) [production]
11:13 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1033, first pool of db1079, db1086, db1094 with low weight (duration: 00m 25s) [production]