|
2020-02-12
ยง
|
| 11:00 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:50 |
<vgutierrez> |
depool cp3050 and reimage as buster - T242093 |
[production] |
| 10:49 |
<vgutierrez> |
pool cp30[51,52] running buster - T242093 |
[production] |
| 10:45 |
<vgutierrez> |
depool cp20[19,25] and reimage as buster - T242093 |
[production] |
| 10:42 |
<vgutierrez> |
pool cp2026 running buster - T242093 |
[production] |
| 10:36 |
<vgutierrez> |
pool cp2023 running buster - T242093 |
[production] |
| 10:34 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:34 |
<moritzm> |
bouncing ferm on ganeti1016, failed to start after boot |
[production] |
| 10:32 |
<vgutierrez> |
Enable KA between ats-tls and varnish-fe on cp4031 - T244464 |
[production] |
| 10:31 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:30 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:29 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:29 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:27 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:25 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:23 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:12 |
<vgutierrez> |
testing trafficserver 8.0.6-rc0 in cp40[26,32] |
[production] |
| 10:06 |
<vgutierrez> |
depool cp20[23,26] and reimage as buster - T242093 |
[production] |
| 10:01 |
<vgutierrez> |
depool cp30[51-52] and reimage as buster - T242093 |
[production] |
| 09:38 |
<ema> |
cp: rolling ats-tls-restart to enable analytics logging T237993 |
[production] |
| 09:26 |
<ema> |
cp4027: ats-tls-restart to enable analytics logging to pipe T237993 |
[production] |
| 09:25 |
<moritzm> |
rolling restart of cassandra on restbase-dev to pick up Java security updates |
[production] |
| 09:17 |
<marostegui> |
Failover m2 master dbproxy from dbproxy1007 to dbproxy1013 - T202367 |
[production] |
| 09:13 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 09:11 |
<marostegui> |
Upgrade and reboot dbproxy1013 before making it master - T202367 |
[production] |
| 08:55 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:46 |
<phedenskog@deploy1001> |
Finished deploy [performance/navtiming@9bbbb58]: (no justification provided) (duration: 00m 05s) |
[production] |
| 08:46 |
<phedenskog@deploy1001> |
Started deploy [performance/navtiming@9bbbb58]: (no justification provided) |
[production] |
| 08:38 |
<marostegui> |
Restart wikibugs as it doesn't show phab comments on irc - T241109 |
[production] |
| 08:21 |
<moritzm> |
installing mesa security updates |
[production] |
| 07:28 |
<vgutierrez> |
pool cp30[53-54] running buster - T242093 |
[production] |
| 07:18 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=30; selector: dc=eqiad,pool=appserver,name=mw132[3-4].* |
[production] |
| 07:16 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=20; selector: dc=eqiad,pool=appserver,service=nginx,name=mw12[3-5].* |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 20 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10391 and previous config saved to /var/cache/conftool/dbconfig/20200212-070250-marostegui.json |
[production] |
| 06:54 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:51 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:50 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:46 |
<marostegui> |
Redact ngwikimedia on db1124:3313 and db2094:3313 T240772 |
[production] |
| 06:22 |
<vgutierrez> |
depool cp30[53-54] and reimage as buster - T242093 |
[production] |
| 06:18 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:17 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:16 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 01:48 |
<XioNoX> |
disabling peering session on cr1-eqsin (they're flapping otherwise) |
[production] |
| 00:44 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/includes/page/ImageHistoryPseudoPager.php: T244937 ImageHistoryPseudoPager: Update doQuery() for IndexPager changes (duration: 01m 03s) |
[production] |
| 00:38 |
<XioNoX> |
reboot cr1-eqsin |
[production] |
| 00:33 |
<XioNoX> |
commit full on cr1-eqsin - T243080 |
[production] |
| 00:21 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings.php: rm wgKartographerIconServer (duration: 01m 02s) |
[production] |
| 00:20 |
<reedy@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: rm wgKartographerIconServer (duration: 01m 03s) |
[production] |