|
2020-08-27
ยง
|
| 10:03 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 09:54 |
<moritzm> |
installing Java security updates on IDP* hosts |
[production] |
| 09:51 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:47 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:45 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 09:44 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 09:44 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 09:43 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 09:43 |
<elukey> |
decommissioning vms schema[12]00[12] (replaced previously by schema[12]00[34] buster vms) |
[production] |
| 09:42 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 09:41 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 09:40 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 09:39 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 09:20 |
<kormat> |
enabling replication from db2105 to db1123 (s3) T243373 |
[production] |
| 09:15 |
<kormat> |
enabling replication from db2079 to db1109 (s8) T243373 |
[production] |
| 09:07 |
<kormat> |
enabling replication from db2090 to db1081 (s4) T243373 |
[production] |
| 08:53 |
<kormat> |
enabling replication from pc2009 to pc1009 (pc3) T243373 |
[production] |
| 08:44 |
<kormat> |
enabling replication from pc2008 to pc1008 (pc2) T243373 |
[production] |
| 08:13 |
<marostegui> |
Enable replication codfw -> eqiad on pc1 T243373 |
[production] |
| 08:01 |
<gehel> |
manual cleanup of stale wdqs deploy crontab on wdqs1009 |
[production] |
| 07:35 |
<marostegui> |
Move pc2010 under pc2007 T243373 |
[production] |
| 07:16 |
<moritzm> |
installing ghostscript security updates on stretch |
[production] |
| 06:50 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 06:49 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:46 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:45 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:34 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:31 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12392 and previous config saved to /var/cache/conftool/dbconfig/20200827-060652-marostegui.json |
[production] |
| 05:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12391 and previous config saved to /var/cache/conftool/dbconfig/20200827-055815-marostegui.json |
[production] |
| 05:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12390 and previous config saved to /var/cache/conftool/dbconfig/20200827-055522-marostegui.json |
[production] |
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1079', diff saved to https://phabricator.wikimedia.org/P12389 and previous config saved to /var/cache/conftool/dbconfig/20200827-055126-marostegui.json |
[production] |
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12388 and previous config saved to /var/cache/conftool/dbconfig/20200827-055104-marostegui.json |
[production] |
| 05:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P12387 and previous config saved to /var/cache/conftool/dbconfig/20200827-054259-marostegui.json |
[production] |
| 05:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1074 db1085 db1078', diff saved to https://phabricator.wikimedia.org/P12386 and previous config saved to /var/cache/conftool/dbconfig/20200827-054114-marostegui.json |
[production] |
| 05:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P12385 and previous config saved to /var/cache/conftool/dbconfig/20200827-053814-marostegui.json |
[production] |
| 05:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12384 and previous config saved to /var/cache/conftool/dbconfig/20200827-053558-marostegui.json |
[production] |
| 05:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1074, db1085, db1079', diff saved to https://phabricator.wikimedia.org/P12383 and previous config saved to /var/cache/conftool/dbconfig/20200827-053509-marostegui.json |
[production] |
| 05:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1074, db1085, db1079', diff saved to https://phabricator.wikimedia.org/P12382 and previous config saved to /var/cache/conftool/dbconfig/20200827-053100-marostegui.json |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P12381 and previous config saved to /var/cache/conftool/dbconfig/20200827-052925-marostegui.json |
[production] |
| 05:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P12380 and previous config saved to /var/cache/conftool/dbconfig/20200827-052818-marostegui.json |
[production] |
| 05:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1074', diff saved to https://phabricator.wikimedia.org/P12379 and previous config saved to /var/cache/conftool/dbconfig/20200827-052413-marostegui.json |
[production] |
| 05:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P12378 and previous config saved to /var/cache/conftool/dbconfig/20200827-051609-marostegui.json |
[production] |
| 05:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12377 and previous config saved to /var/cache/conftool/dbconfig/20200827-051546-marostegui.json |
[production] |
| 05:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12376 and previous config saved to /var/cache/conftool/dbconfig/20200827-050754-marostegui.json |
[production] |
| 05:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1085', diff saved to https://phabricator.wikimedia.org/P12375 and previous config saved to /var/cache/conftool/dbconfig/20200827-050727-marostegui.json |
[production] |
| 04:53 |
<marostegui> |
Stop db1074 and db2107 in sync to fix drifts on s2 change_tag - T260042 |
[production] |
| 04:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074', diff saved to https://phabricator.wikimedia.org/P12374 and previous config saved to /var/cache/conftool/dbconfig/20200827-045329-marostegui.json |
[production] |
| 04:04 |
<ryankemper@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1006.wikimedia.org |
[production] |
| 04:03 |
<ryankemper@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1005.wikimedia.org |
[production] |