51-100 of 10000 results (36ms)
2018-10-09 §
10:47 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) [production]
10:41 <START> - Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) [production]
10:40 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@neodymium) [production]
10:40 <START> - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@neodymium) [production]
10:37 <END> (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@neodymium) [production]
10:36 <START> - Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@neodymium) [production]
10:35 <onimisionipe> deploying prometheus-blazegraph-exporter 0.6 on all wdqs clusters - T206123 [production]
10:34 <volans> about to perform live-test of the inverted switchdc (eqiad->codfw), actions will be real but basically noop due to codfw being already active - T203777 [production]
09:25 <elukey> swapped Hadoop's hive/oozie from analytics1003 to an-coord1001 [production]
09:16 <ema> restart pybal on lvs1005 to pick up config changes (conf2001 -> conf1004) [production]
09:00 <ema> re-enable puppet/pybal on lvs1002, IPv6 connectivity with phab1001 working again T201039 [production]
08:16 <elukey> update puppet compiler facts [production]
08:06 <onimisionipe> depooling wdqs2001 to catch up on lag -T206423 [production]
07:03 <akosiaris> restart zuul and zuul-merger on contint1001 for the upgrade of zuul to finish [production]
06:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1122 (duration: 00m 57s) [production]
05:19 <marostegui> Stop MySQL on db1122 for binlog format change, mysql and kernel upgrade [production]
05:14 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1122 (duration: 00m 59s) [production]
02:41 <krinkle@deploy1001> Synchronized wmf-config/profiler.php: T176916 / T206092 - Ie86e88777c48 (duration: 00m 56s) [production]
02:21 <krinkle@deploy1001> Synchronized wmf-config/arclamp.php: T176916 - Id79baae90: ensure file exists before Ie86e88777c48 (duration: 00m 57s) [production]
00:04 <krinkle@deploy1001> Synchronized php-1.32.0-wmf.24/includes/libs/rdbms/database: T201900 - I8ae754a2518 (duration: 00m 59s) [production]
2018-10-08 §
22:45 <XioNoX> increase accepted-prefix-limit for 24115 on cr4-ulsfo [production]
22:41 <XioNoX> clear BGP neighbor cr1-eqsin:AS9583 (bgp limit threshold reached) [production]
21:11 <ejegg> updated payments-wiki from d623de9494 to 1472604b6e [production]
20:42 <gehel> repooling wdqs2003 catched up on lag - T206423 [production]
19:41 <XioNoX> troubleshooting asw2-b-eqid with JTAC - T201039 [production]
19:08 <gehel> depooling wdqs2003 to catch up on lag -T206423 [production]
19:00 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465194|Enable MCR read-new mode on some small wikis (T198308)]] (duration: 00m 56s) [production]
18:55 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@bd698bd]: WDQS deployment - New federation whitelist entries (duration: 10m 07s) [production]
18:45 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@bd698bd]: WDQS deployment - New federation whitelist entries [production]
18:37 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@bd698bd]: WDQS test deployment - New federation whitelist entries(wdqs1009) (duration: 00m 33s) [production]
18:37 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@bd698bd]: WDQS test deployment - New federation whitelist entries(wdqs1009) [production]
18:36 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465176|Enable Extension:File exporter to mrwikipedia (T206437)]] (duration: 00m 57s) [production]
16:29 <XioNoX> push firewall filter counters on asw2-b-eqiad - T201039 [production]
16:28 <elukey> restart eventlogging on eventlog1002 for python security upgrades [production]
14:05 <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 57s) [production]
14:03 <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 56s) [production]
13:43 <elukey> restart confd on esams nodes to pick up new srv settings [production]
13:41 <elukey> restart navtiming.service on webperf1001 to pick up the dns change for etcd [production]
13:39 <marostegui> Enable gtid on the following slaves: db2068 db1122 db1117:3323 [production]
13:37 <elukey> restart confd on all the other eqiad nodes to pick up new srv records [production]
13:32 <elukey> restart confd on cp1* to pick up new srv records [production]
13:11 <_joe_> purging the dnsrec cache for eqiad,esams etcd client SRV records [production]
13:09 <ema> depool eqiad front-edge traffic T201039 [production]
13:05 <banyek> converting cebwiki.templatelinks to TokuDB on host dbstore1002.eqiad.wmnet (T205544) [production]
13:04 <banyek> downtime notifications for dbstore1002 repliaction threads (T205544) [production]
12:49 <banyek> pt-kill-wmf enabled on the wikireplicas (T203674) [production]
11:59 <_joe_> restart pybal in esams, after running puppet, to switch etcd cluster used [production]
11:46 <_joe_> restart pybal on lvs1001 [production]
11:46 <addshore> SWAT done [production]
11:45 <addshore@deploy1001> Synchronized wmf-config/throttle.php: Add throttle exception for Netherlands Hackathon October 2018 - Wiki Techstorm T206241, and remove other rules. (duration: 00m 56s) [production]