|
2022-03-17
ยง
|
| 17:16 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1018.eqiad.wmnet with reason: host reimage |
[production] |
| 17:16 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 17:16 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 17:15 |
<dancy@deploy1002> |
Synchronized README: testing mediawiki image build (duration: 02m 11s) |
[production] |
| 17:11 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 17:10 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1006.eqiad.wmnet with OS bullseye |
[production] |
| 17:09 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubernetes1020.eqiad.wmnet with OS bullseye |
[production] |
| 17:09 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1020.eqiad.wmnet with OS bullseye |
[production] |
| 17:09 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubernetes1022.eqiad.wmnet with OS bullseye |
[production] |
| 17:08 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1022.eqiad.wmnet with OS bullseye |
[production] |
| 17:08 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubernetes1021.eqiad.wmnet with OS bullseye |
[production] |
| 17:08 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1021.eqiad.wmnet with OS bullseye |
[production] |
| 17:07 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubernetes1019.eqiad.wmnet with OS bullseye |
[production] |
| 17:06 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1019.eqiad.wmnet with OS bullseye |
[production] |
| 17:06 |
<bblack> |
geodns - Cyprus routed to new drmrs edge DC (first live users!) - will phase in over the standard 10 minute DNS TTL |
[production] |
| 17:05 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubernetes1018.eqiad.wmnet with OS bullseye |
[production] |
| 17:04 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes1018.eqiad.wmnet with OS bullseye |
[production] |
| 17:03 |
<volans> |
restart atftp on install1003 |
[production] |
| 17:01 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 17:00 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 17:00 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 16:50 |
<dancy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 16:48 |
<XioNoX> |
disable BGP to Lumen in codfw for fiber move |
[production] |
| 16:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T298557)', diff saved to https://phabricator.wikimedia.org/P22794 and previous config saved to /var/cache/conftool/dbconfig/20220317-164228-marostegui.json |
[production] |
| 16:42 |
<robh@cumin1001> |
START - Cookbook sre.hosts.reimage for host dumpsdata1006.eqiad.wmnet with OS bullseye |
[production] |
| 16:42 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:40 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:36 |
<moritzm> |
restarting LDAP replicas for openssl update |
[production] |
| 16:35 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudcephmon1003.eqiad.wmnet on all recursors |
[production] |
| 16:35 |
<dcaro@cumin1001> |
START - Cookbook sre.dns.wipe-cache cloudcephmon1003.eqiad.wmnet on all recursors |
[production] |
| 16:35 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudcephmon1003.eqiad.wmnet on all recursors |
[production] |
| 16:35 |
<jbond@cumin1001> |
START - Cookbook sre.dns.wipe-cache cloudcephmon1003.eqiad.wmnet on all recursors |
[production] |
| 16:34 |
<ryankemper> |
[WDQS] Pooled `wdqs2001` (caught up on lag) |
[production] |
| 16:31 |
<andrewbogott> |
sudo service networking restart on puppetmaster1003 |
[production] |
| 16:28 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 16:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P22793 and previous config saved to /var/cache/conftool/dbconfig/20220317-162723-marostegui.json |
[production] |
| 16:15 |
<robh@cumin1001> |
START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 16:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P22792 and previous config saved to /var/cache/conftool/dbconfig/20220317-161218-marostegui.json |
[production] |
| 16:11 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 16:10 |
<XioNoX> |
pfw3-codfw move traffic to cr2 uplink |
[production] |
| 16:05 |
<oblivian@puppetmaster1001> |
conftool action : edit; selector: name=random_q |
[production] |
| 16:04 |
<ryankemper> |
[WDQS] Depooled `wdqs2001` (~4.85 hours of lag to catch up) |
[production] |
| 16:03 |
<ryankemper> |
[WDQS] `ryankemper@wdqs2001:~$ sudo systemctl restart wdqs-blazegraph.service` |
[production] |
| 16:03 |
<ryankemper> |
[WDQS] Pooled `wdqs2003` (caught up on lag) |
[production] |
| 16:00 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:00 |
<robh@cumin1001> |
START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 16:00 |
<moritzm> |
restarting apache on logstash* |
[production] |
| 15:57 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 60980ce85c080fadaf0b2cb561be53f861ca94e0: ptwiki: Disable Growth image recommendation (T302828) (duration: 00m 53s) |
[production] |
| 15:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T298557)', diff saved to https://phabricator.wikimedia.org/P22790 and previous config saved to /var/cache/conftool/dbconfig/20220317-155713-marostegui.json |
[production] |
| 15:49 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |