|
2020-03-31
ยง
|
| 17:38 |
<elukey> |
restart elasticsearch_6@cloudelastic-chi-eqiad.service on cloudelastic1001 to see if it recovers from a trashing/gc state - T231517 |
[production] |
| 16:30 |
<marxarelli> |
1.35.0-wmf.26 was branched at bec758b668aaa57fc259a1d0ecf3b35340d2661b for T247773 |
[production] |
| 16:24 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 01m 00s) |
[production] |
| 16:15 |
<vgutierrez> |
pool cp2037 - T248816 |
[production] |
| 15:39 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:36 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:35 |
<mutante> |
decom mw1254 through mw1258 (last remaining old servers in rack D5, depooled a while ago and average response time is again under 200ms) T247780 |
[production] |
| 15:33 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:29 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:29 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:28 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:27 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:27 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:27 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:26 |
<vgutierrez> |
depool & decommission cp2010 - T249002 |
[production] |
| 15:15 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 58s) |
[production] |
| 15:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T245794 Enable DiscussionTools as a beta feature on four wikis (duration: 01m 00s) |
[production] |
| 15:05 |
<cdanis> |
cr1-eqiad: commit flex-flow-sizing T248394 |
[production] |
| 15:01 |
<cdanis> |
cr2-eqiad: commit flex-flow-sizing T248394 |
[production] |
| 14:43 |
<vgutierrez> |
pool cp2036 - T248816 |
[production] |
| 14:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw125[4-8].eqiad.wmnet |
[production] |
| 14:20 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:20 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:19 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:17 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1091 after schema change', diff saved to https://phabricator.wikimedia.org/P10834 and previous config saved to /var/cache/conftool/dbconfig/20200331-141459-marostegui.json |
[production] |
| 14:10 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:10 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:05 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw125[4-8].eqiad.wmnet |
[production] |
| 13:31 |
<vgutierrez> |
Enable TLS Session tickets in eqsin - T245616 |
[production] |
| 13:05 |
<XioNoX> |
update nat on pfw3-codfw - T248906 |
[production] |
| 13:03 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:03 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:49 |
<_joe_> |
switching all appserver canaries to envoy |
[production] |
| 12:46 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:45 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:45 |
<marostegui> |
Deploy schema change on db1091 |
[production] |
| 12:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1091 for schema change', diff saved to https://phabricator.wikimedia.org/P10833 and previous config saved to /var/cache/conftool/dbconfig/20200331-124452-marostegui.json |
[production] |
| 12:34 |
<_joe_> |
transitioning mw1261 to envoy |
[production] |
| 12:23 |
<vgutierrez> |
rolling upgrade of ATS to version 8.0.6-1wm5 - T248938 |
[production] |
| 11:30 |
<Lucas_WMDE> |
EU SWAT done |
[production] |
| 11:30 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:584874|Disable TwoColConflict talk page workflow (T230231)]], take II (duration: 00m 57s) |
[production] |
| 11:29 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:584874|Disable TwoColConflict talk page workflow (T230231)]] (duration: 00m 58s) |
[production] |
| 11:11 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:584574|Enable ContentTranslation in Lithuanian Wikipedia as a default tool (T248179)]], take II (duration: 00m 59s) |
[production] |
| 11:10 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:584574|Enable ContentTranslation in Lithuanian Wikipedia as a default tool (T248179)]] (duration: 01m 00s) |
[production] |
| 10:46 |
<_joe_> |
disabled puppet on canary appservers, potentially dangerous change ahead |
[production] |
| 10:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 after schema change', diff saved to https://phabricator.wikimedia.org/P10831 and previous config saved to /var/cache/conftool/dbconfig/20200331-101953-marostegui.json |
[production] |
| 10:03 |
<XioNoX> |
add BGP to AS41327 in AMS-IX |
[production] |
| 09:49 |
<XioNoX> |
push homer diffs to mr1-eqsin |
[production] |
| 09:36 |
<XioNoX> |
push homer diffs to mr1-eqiad |
[production] |