|
2020-09-17
ยง
|
| 16:25 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:21 |
<marostegui> |
Restart wikibugs |
[production] |
| 16:18 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:15 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:15 |
<papaul> |
replacing msw-d8-codfw |
[production] |
| 16:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change db1131 IP after moving it to a different rack T262901', diff saved to https://phabricator.wikimedia.org/P12639 and previous config saved to /var/cache/conftool/dbconfig/20200917-160540-marostegui.json |
[production] |
| 16:03 |
<marostegui> |
Recreate db1131 on tendril T262901 |
[production] |
| 15:59 |
<marostegui> |
Update rack location on zarcillo for db1131 T262901 |
[production] |
| 15:57 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2114: repool at 100% T259831', diff saved to https://phabricator.wikimedia.org/P12638 and previous config saved to /var/cache/conftool/dbconfig/20200917-155708-kormat.json |
[production] |
| 15:44 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2114: repool at 75% T259831', diff saved to https://phabricator.wikimedia.org/P12637 and previous config saved to /var/cache/conftool/dbconfig/20200917-154431-kormat.json |
[production] |
| 15:43 |
<mepps> |
updated payments-wiki from 3c073a6a56 to 1e5a52ed26 |
[production] |
| 15:35 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2114: repool at 50% T259831', diff saved to https://phabricator.wikimedia.org/P12636 and previous config saved to /var/cache/conftool/dbconfig/20200917-153514-kormat.json |
[production] |
| 15:25 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:20 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2114: repool at 25% T259831', diff saved to https://phabricator.wikimedia.org/P12635 and previous config saved to /var/cache/conftool/dbconfig/20200917-152019-kormat.json |
[production] |
| 15:17 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2125 T260670', diff saved to https://phabricator.wikimedia.org/P12634 and previous config saved to /var/cache/conftool/dbconfig/20200917-151347-marostegui.json |
[production] |
| 15:06 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 15:04 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2125 T260670', diff saved to https://phabricator.wikimedia.org/P12633 and previous config saved to /var/cache/conftool/dbconfig/20200917-150234-marostegui.json |
[production] |
| 15:02 |
<jynus> |
deploying extended grants for admin account on sys/p_s at s8@codfw T195578 |
[production] |
| 15:00 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 15:00 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 14:55 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:54 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db2114: depool for schema change T259831', diff saved to https://phabricator.wikimedia.org/P12632 and previous config saved to /var/cache/conftool/dbconfig/20200917-145451-kormat.json |
[production] |
| 14:49 |
<cmjohnson1> |
ending pdu maintenance in eqiad |
[production] |
| 14:40 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 14:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2125 T260670', diff saved to https://phabricator.wikimedia.org/P12631 and previous config saved to /var/cache/conftool/dbconfig/20200917-143914-marostegui.json |
[production] |
| 14:32 |
<papaul> |
replacing msw-d1,d2,d3,d4,d5 and d6 |
[production] |
| 14:31 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 14:28 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 14:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2125 T260670', diff saved to https://phabricator.wikimedia.org/P12630 and previous config saved to /var/cache/conftool/dbconfig/20200917-141825-marostegui.json |
[production] |
| 14:02 |
<marostegui> |
Start mysql on db1125 after PDU maintenance T261459 |
[production] |
| 14:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2125 T260670', diff saved to https://phabricator.wikimedia.org/P12629 and previous config saved to /var/cache/conftool/dbconfig/20200917-140014-marostegui.json |
[production] |
| 13:33 |
<jayme> |
ran ipvsadm -D -t 10.2.2.14:8888 on lvs1016.eqiad.wmnet,lvs1015.eqiad.wmnet |
[production] |
| 13:33 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 13:33 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 13:32 |
<jayme> |
ran ipvsadm -D -t 10.2.2.31:8748 on lvs1016.eqiad.wmnet,lvs1015.eqiad.wmnet |
[production] |
| 13:32 |
<jayme> |
ran ipvsadm -D -t 10.2.1.31:8748 on lvs2010.codfw.wmnet,lvs2009.codfw.wmnet |
[production] |
| 13:32 |
<jayme> |
ran ipvsadm -D -t 10.2.1.14:8888 on lvs2010.codfw.wmnet,lvs2009.codfw.wmnet |
[production] |
| 13:25 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Start depooling db2114 T259831', diff saved to https://phabricator.wikimedia.org/P12628 and previous config saved to /var/cache/conftool/dbconfig/20200917-132513-kormat.json |
[production] |
| 13:20 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 13:20 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 13:19 |
<jayme> |
restarting pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet |
[production] |
| 13:17 |
<marostegui> |
Stop MySQL on db2125 for on-site maintenance T260670 |
[production] |
| 13:14 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 13:14 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 13:13 |
<jayme> |
restarting pybal on lvs1016.eqiad.wmnet,lvs2010.codfw.wmnet |
[production] |
| 13:03 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.9 |
[production] |
| 12:18 |
<cmjohnson1> |
pdu swap maintenance beginning now for racks D1, D2 and C1 eqiad |
[production] |