|
2020-02-12
§
|
| 10:25 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:23 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:12 |
<vgutierrez> |
testing trafficserver 8.0.6-rc0 in cp40[26,32] |
[production] |
| 10:06 |
<vgutierrez> |
depool cp20[23,26] and reimage as buster - T242093 |
[production] |
| 10:01 |
<vgutierrez> |
depool cp30[51-52] and reimage as buster - T242093 |
[production] |
| 09:38 |
<ema> |
cp: rolling ats-tls-restart to enable analytics logging T237993 |
[production] |
| 09:26 |
<ema> |
cp4027: ats-tls-restart to enable analytics logging to pipe T237993 |
[production] |
| 09:25 |
<moritzm> |
rolling restart of cassandra on restbase-dev to pick up Java security updates |
[production] |
| 09:17 |
<marostegui> |
Failover m2 master dbproxy from dbproxy1007 to dbproxy1013 - T202367 |
[production] |
| 09:13 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 09:11 |
<marostegui> |
Upgrade and reboot dbproxy1013 before making it master - T202367 |
[production] |
| 08:55 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:46 |
<phedenskog@deploy1001> |
Finished deploy [performance/navtiming@9bbbb58]: (no justification provided) (duration: 00m 05s) |
[production] |
| 08:46 |
<phedenskog@deploy1001> |
Started deploy [performance/navtiming@9bbbb58]: (no justification provided) |
[production] |
| 08:38 |
<marostegui> |
Restart wikibugs as it doesn't show phab comments on irc - T241109 |
[production] |
| 08:21 |
<moritzm> |
installing mesa security updates |
[production] |
| 07:28 |
<vgutierrez> |
pool cp30[53-54] running buster - T242093 |
[production] |
| 07:18 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=30; selector: dc=eqiad,pool=appserver,name=mw132[3-4].* |
[production] |
| 07:16 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=20; selector: dc=eqiad,pool=appserver,service=nginx,name=mw12[3-5].* |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 20 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10391 and previous config saved to /var/cache/conftool/dbconfig/20200212-070250-marostegui.json |
[production] |
| 06:54 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:51 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:50 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:46 |
<marostegui> |
Redact ngwikimedia on db1124:3313 and db2094:3313 T240772 |
[production] |
| 06:22 |
<vgutierrez> |
depool cp30[53-54] and reimage as buster - T242093 |
[production] |
| 06:18 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:17 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:16 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 01:48 |
<XioNoX> |
disabling peering session on cr1-eqsin (they're flapping otherwise) |
[production] |
| 00:44 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/includes/page/ImageHistoryPseudoPager.php: T244937 ImageHistoryPseudoPager: Update doQuery() for IndexPager changes (duration: 01m 03s) |
[production] |
| 00:38 |
<XioNoX> |
reboot cr1-eqsin |
[production] |
| 00:33 |
<XioNoX> |
commit full on cr1-eqsin - T243080 |
[production] |
| 00:21 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings.php: rm wgKartographerIconServer (duration: 01m 02s) |
[production] |
| 00:20 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: rm wgKartographerIconServer (duration: 01m 03s) |
[production] |
| 00:16 |
<eileen> |
civicrm revision changed from ee9edf8137 to 55b2afb6eb, config revision is 561ae21f77 |
[production] |
|
2020-02-11
§
|
| 22:04 |
<XioNoX> |
switchover RE mastership back re0 on cr1-eqsin - T243080 |
[production] |
| 21:50 |
<XioNoX> |
reboot re0:cr1-eqsin (backup) - T243080 |
[production] |
| 21:45 |
<cdanis> |
repool eqiad |
[production] |
| 21:37 |
<bblack@cumin1001> |
conftool action : set/pooled=yes; selector: name=^cp107.* |
[production] |
| 21:36 |
<bblack@cumin1001> |
conftool action : set/pooled=yes; selector: name=^cp108.* |
[production] |
| 21:36 |
<bblack> |
re-pooling all cp10xx in eqiad |
[production] |
| 21:32 |
<XioNoX> |
switchover RE mastership on cr1-eqsin - T243080 |
[production] |
| 21:13 |
<robh> |
cp1067 powered back into service post firmware update via T243167 |
[production] |
| 21:11 |
<cdanis> |
depool eqiad |
[production] |
| 21:01 |
<marxarelli> |
completed group0 to 1.35.0-wmf.19 (T233867) |
[production] |
| 20:57 |
<robh> |
cp108[45] returned to service, depooling cp108[67]for firmware update via T243167 |
[production] |
| 20:54 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.35.0-wmf.19 |
[production] |
| 20:53 |
<mutante> |
gerrit - moving gerrit db_pass from private module passwords to private hieradata |
[production] |