5101-5150 of 10000 results (50ms)
2018-02-19 §
14:59 <jynus> testing new dbproxy1010 configuration locally to pool labsdb1010 for analytics [production]
13:44 <godog> roll-restart prometheus after retention period bump [production]
13:19 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1063 (duration: 00m 55s) [production]
13:19 <marostegui> Deploy schema change on dbstore1002 - T187089 T185128 T153182 [production]
13:16 <ema> upgrade cache_text@eqsin to varnish 5 [production]
12:48 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully repool db1098 s6 and s7 (duration: 00m 55s) [production]
12:27 <marostegui> Deploy schema change on db1063 - T187089 T185128 T153182 [production]
12:27 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1063 for alter table (duration: 00m 55s) [production]
12:17 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1093 (duration: 00m 55s) [production]
12:14 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098 s6 and s7 (duration: 00m 56s) [production]
11:07 <jdrewniak@tin> Synchronized portals: Wikimedia portals Update: [[gerrit:412657|Bumping portals to master (T128546)]] (duration: 00m 57s) [production]
11:06 <jdrewniak@tin> Synchronized portals/prod/wikipedia.org/assets: Wikimedia portals Update: [[gerrit:412657|Bumping portals to master (T128546)]] (duration: 00m 56s) [production]
10:45 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098 s6 and s7 (duration: 00m 55s) [production]
10:35 <marostegui> Deploy schema change on db1093 - T187089 T185128 T153182 [production]
10:35 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 56s) [production]
10:27 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1098 s6 and s7 (duration: 00m 56s) [production]
10:10 <marostegui> Upgrade mariadb and kernel on db1098 [production]
09:59 <marostegui> Enable GTID on dbstore2002:3313 and dbstore2001:3316 [production]
09:57 <marostegui> Enable GTID on dbstore2002 and dbstore2001 for x1 [production]
09:55 <jynus> reenable gtid replication on db1053 and db2042 [production]
09:53 <jmm@puppetmaster1001> conftool action : set/pooled=inactive; selector: mw1260.eqiad.wmnet [production]
09:53 <jmm@puppetmaster1001> conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet [production]
09:43 <marostegui> Upgrade mariadb and kernel on db2033 [production]
09:36 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully repool db1090 (duration: 00m 55s) [production]
09:18 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1105 - T162807 (duration: 00m 55s) [production]
08:55 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 for mariadb and kernel upgrade (duration: 00m 55s) [production]
08:49 <marostegui> Deploy schema change on db1098:3316 - T187089 T185128 T153182 [production]
08:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 55s) [production]
08:24 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1090 (duration: 00m 55s) [production]
08:11 <godog> repool mw1227 - T149287 [production]
08:02 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Promote db2034 to x1 codfw master - T184888 (duration: 00m 56s) [production]
07:58 <moritzm> installing werkzeug security updates on trusty [production]
07:42 <marostegui> Change topology on x1 codfw - T184888 [production]
07:21 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1090 (duration: 00m 55s) [production]
07:01 <marostegui> Reboot db1090 for kernel ugprade, mariadb upgrade, socket path location upgrade [production]
07:00 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1090 (duration: 00m 55s) [production]
06:44 <marostegui> Stop MySQL on db1089 to update its socket path [production]
06:42 <marostegui> Deploy schema change on s6 codfw master (db2039), this will generate lag on codfw - T187089 T185128 T153182 [production]
06:40 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089 and db1105 - T162807 (duration: 00m 56s) [production]
05:29 <andrew@tin> Finished deploy [horizon/deploy@6a40f84]: rolling out several horizon bugfixes (duration: 03m 14s) [production]
05:26 <andrew@tin> Started deploy [horizon/deploy@6a40f84]: rolling out several horizon bugfixes [production]
02:50 <l10nupdate@tin> scap sync-l10n completed (1.31.0-wmf.21) (duration: 10m 59s) [production]
2018-02-18 §
15:49 <_joe_> rolling restart (1 at a time, staggered by 2 minutes) of 18 api appservers in equiad [production]
2018-02-17 §
17:33 <twentyafterfour> restarting apache on phab1001 to clear deadlocked workers. refs T182832 [production]
03:15 <demon@tin> Pruned MediaWiki: 1.31.0-wmf.20 [keeping static files] (duration: 01m 17s) [production]
03:12 <demon@tin> Pruned MediaWiki: 1.31.0-wmf.17 (duration: 04m 32s) [production]
2018-02-16 §
21:12 <hashar> Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3 | T187567 [production]
20:40 <andrew@tin> Finished deploy [horizon/deploy@efcba2b]: sudo dashboard update (duration: 01m 16s) [production]
20:39 <andrew@tin> Started deploy [horizon/deploy@efcba2b]: sudo dashboard update [production]
20:11 <andrew@tin> Finished deploy [horizon/deploy@1fdd122]: two more small fixes (duration: 01m 21s) [production]