|
2018-10-09
§
|
| 10:53 |
<START> |
- Cookbook sre.switchdc.mediawiki.01-stop-maintenance (volans@neodymium) |
[production] |
| 10:51 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) |
[production] |
| 10:49 |
<onimisionipe> |
repooling wdqs2001 catched up on lag - T206423 |
[production] |
| 10:48 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) |
[production] |
| 10:47 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) |
[production] |
| 10:41 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) |
[production] |
| 10:40 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@neodymium) |
[production] |
| 10:40 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@neodymium) |
[production] |
| 10:37 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@neodymium) |
[production] |
| 10:36 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@neodymium) |
[production] |
| 10:35 |
<onimisionipe> |
deploying prometheus-blazegraph-exporter 0.6 on all wdqs clusters - T206123 |
[production] |
| 10:34 |
<volans> |
about to perform live-test of the inverted switchdc (eqiad->codfw), actions will be real but basically noop due to codfw being already active - T203777 |
[production] |
| 09:25 |
<elukey> |
swapped Hadoop's hive/oozie from analytics1003 to an-coord1001 |
[production] |
| 09:16 |
<ema> |
restart pybal on lvs1005 to pick up config changes (conf2001 -> conf1004) |
[production] |
| 09:00 |
<ema> |
re-enable puppet/pybal on lvs1002, IPv6 connectivity with phab1001 working again T201039 |
[production] |
| 08:16 |
<elukey> |
update puppet compiler facts |
[production] |
| 08:06 |
<onimisionipe> |
depooling wdqs2001 to catch up on lag -T206423 |
[production] |
| 07:03 |
<akosiaris> |
restart zuul and zuul-merger on contint1001 for the upgrade of zuul to finish |
[production] |
| 06:37 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1122 (duration: 00m 57s) |
[production] |
| 05:19 |
<marostegui> |
Stop MySQL on db1122 for binlog format change, mysql and kernel upgrade |
[production] |
| 05:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1122 (duration: 00m 59s) |
[production] |
| 02:41 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: T176916 / T206092 - Ie86e88777c48 (duration: 00m 56s) |
[production] |
| 02:21 |
<krinkle@deploy1001> |
Synchronized wmf-config/arclamp.php: T176916 - Id79baae90: ensure file exists before Ie86e88777c48 (duration: 00m 57s) |
[production] |
| 00:04 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.24/includes/libs/rdbms/database: T201900 - I8ae754a2518 (duration: 00m 59s) |
[production] |
|
2018-10-08
§
|
| 22:45 |
<XioNoX> |
increase accepted-prefix-limit for 24115 on cr4-ulsfo |
[production] |
| 22:41 |
<XioNoX> |
clear BGP neighbor cr1-eqsin:AS9583 (bgp limit threshold reached) |
[production] |
| 21:11 |
<ejegg> |
updated payments-wiki from d623de9494 to 1472604b6e |
[production] |
| 20:42 |
<gehel> |
repooling wdqs2003 catched up on lag - T206423 |
[production] |
| 19:41 |
<XioNoX> |
troubleshooting asw2-b-eqid with JTAC - T201039 |
[production] |
| 19:08 |
<gehel> |
depooling wdqs2003 to catch up on lag -T206423 |
[production] |
| 19:00 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465194|Enable MCR read-new mode on some small wikis (T198308)]] (duration: 00m 56s) |
[production] |
| 18:55 |
<onimisionipe@deploy1001> |
Finished deploy [wdqs/wdqs@bd698bd]: WDQS deployment - New federation whitelist entries (duration: 10m 07s) |
[production] |
| 18:45 |
<onimisionipe@deploy1001> |
Started deploy [wdqs/wdqs@bd698bd]: WDQS deployment - New federation whitelist entries |
[production] |
| 18:37 |
<onimisionipe@deploy1001> |
Finished deploy [wdqs/wdqs@bd698bd]: WDQS test deployment - New federation whitelist entries(wdqs1009) (duration: 00m 33s) |
[production] |
| 18:37 |
<onimisionipe@deploy1001> |
Started deploy [wdqs/wdqs@bd698bd]: WDQS test deployment - New federation whitelist entries(wdqs1009) |
[production] |
| 18:36 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465176|Enable Extension:File exporter to mrwikipedia (T206437)]] (duration: 00m 57s) |
[production] |
| 16:29 |
<XioNoX> |
push firewall filter counters on asw2-b-eqiad - T201039 |
[production] |
| 16:28 |
<elukey> |
restart eventlogging on eventlog1002 for python security upgrades |
[production] |
| 14:05 |
<banyek@deploy1001> |
Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 57s) |
[production] |
| 14:03 |
<banyek@deploy1001> |
Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 56s) |
[production] |
| 13:43 |
<elukey> |
restart confd on esams nodes to pick up new srv settings |
[production] |
| 13:41 |
<elukey> |
restart navtiming.service on webperf1001 to pick up the dns change for etcd |
[production] |
| 13:39 |
<marostegui> |
Enable gtid on the following slaves: db2068 db1122 db1117:3323 |
[production] |
| 13:37 |
<elukey> |
restart confd on all the other eqiad nodes to pick up new srv records |
[production] |
| 13:32 |
<elukey> |
restart confd on cp1* to pick up new srv records |
[production] |
| 13:11 |
<_joe_> |
purging the dnsrec cache for eqiad,esams etcd client SRV records |
[production] |
| 13:09 |
<ema> |
depool eqiad front-edge traffic T201039 |
[production] |
| 13:05 |
<banyek> |
converting cebwiki.templatelinks to TokuDB on host dbstore1002.eqiad.wmnet (T205544) |
[production] |
| 13:04 |
<banyek> |
downtime notifications for dbstore1002 repliaction threads (T205544) |
[production] |
| 12:49 |
<banyek> |
pt-kill-wmf enabled on the wikireplicas (T203674) |
[production] |