|
2020-03-31
ยง
|
| 13:03 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:49 |
<_joe_> |
switching all appserver canaries to envoy |
[production] |
| 12:46 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:45 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:45 |
<marostegui> |
Deploy schema change on db1091 |
[production] |
| 12:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1091 for schema change', diff saved to https://phabricator.wikimedia.org/P10833 and previous config saved to /var/cache/conftool/dbconfig/20200331-124452-marostegui.json |
[production] |
| 12:34 |
<_joe_> |
transitioning mw1261 to envoy |
[production] |
| 12:23 |
<vgutierrez> |
rolling upgrade of ATS to version 8.0.6-1wm5 - T248938 |
[production] |
| 11:30 |
<Lucas_WMDE> |
EU SWAT done |
[production] |
| 11:30 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:584874|Disable TwoColConflict talk page workflow (T230231)]], take II (duration: 00m 57s) |
[production] |
| 11:29 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:584874|Disable TwoColConflict talk page workflow (T230231)]] (duration: 00m 58s) |
[production] |
| 11:11 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:584574|Enable ContentTranslation in Lithuanian Wikipedia as a default tool (T248179)]], take II (duration: 00m 59s) |
[production] |
| 11:10 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:584574|Enable ContentTranslation in Lithuanian Wikipedia as a default tool (T248179)]] (duration: 01m 00s) |
[production] |
| 10:46 |
<_joe_> |
disabled puppet on canary appservers, potentially dangerous change ahead |
[production] |
| 10:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 after schema change', diff saved to https://phabricator.wikimedia.org/P10831 and previous config saved to /var/cache/conftool/dbconfig/20200331-101953-marostegui.json |
[production] |
| 10:03 |
<XioNoX> |
add BGP to AS41327 in AMS-IX |
[production] |
| 09:49 |
<XioNoX> |
push homer diffs to mr1-eqsin |
[production] |
| 09:36 |
<XioNoX> |
push homer diffs to mr1-eqiad |
[production] |
| 09:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:10 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 09:09 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 09:05 |
<vgutierrez> |
upload trafficserver 8.0.5-1wm6 to apt.wm.o (buster) - T248938 |
[production] |
| 09:00 |
<vgutierrez> |
depool & decommission cp2011 - T248950 |
[production] |
| 08:52 |
<marostegui> |
Stop MySQL on db2093 for reimage to buster |
[production] |
| 08:44 |
<vgutierrez> |
pool cp2035 - T248816 |
[production] |
| 08:31 |
<mutante> |
signed puppet cert for planet1002.eqiad.wmnet |
[production] |
| 08:29 |
<marostegui> |
Depool db1084 for schema change |
[production] |
| 08:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1084 for schema change', diff saved to https://phabricator.wikimedia.org/P10829 and previous config saved to /var/cache/conftool/dbconfig/20200331-082904-marostegui.json |
[production] |
| 08:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1081 after schema change', diff saved to https://phabricator.wikimedia.org/P10828 and previous config saved to /var/cache/conftool/dbconfig/20200331-082711-marostegui.json |
[production] |
| 08:17 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 08:08 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:01 |
<XioNoX> |
delete unused ROA for ARIN v4 prefixes - T235886 |
[production] |
| 07:49 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 07:49 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:17 |
<vgutierrez> |
pool cp2034 - T248816 |
[production] |
| 07:16 |
<marostegui> |
Deploy schema change on db1081 |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1081 for schema change', diff saved to https://phabricator.wikimedia.org/P10827 and previous config saved to /var/cache/conftool/dbconfig/20200331-071547-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314 after schema change', diff saved to https://phabricator.wikimedia.org/P10826 and previous config saved to /var/cache/conftool/dbconfig/20200331-071401-marostegui.json |
[production] |
| 06:48 |
<marostegui> |
Deploy schema change on db1103:3314 |
[production] |
| 06:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3314 for schema change', diff saved to https://phabricator.wikimedia.org/P10825 and previous config saved to /var/cache/conftool/dbconfig/20200331-064707-marostegui.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1097:3314 after schema change', diff saved to https://phabricator.wikimedia.org/P10824 and previous config saved to /var/cache/conftool/dbconfig/20200331-064627-marostegui.json |
[production] |
| 05:55 |
<marostegui> |
Drop nova and nova_api from m5 master (db1133) - T248313 |
[production] |
| 05:55 |
<kart_> |
Updated cxserver to 2020-03-30-145349-production (T248578) |
[production] |
| 05:55 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 05:54 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:53 |
<vgutierrez> |
depool && decommission cp2007 - T248941 |
[production] |
| 05:51 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:48 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:46 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |