|
2023-08-23
ยง
|
| 11:21 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.wipe-cache atlas2001.wikimedia.org on all recursors |
[production] |
| 11:21 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 11:21 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM atlas2001.wikimedia.org - ayounsi@cumin1001" |
[production] |
| 11:19 |
<ayounsi@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM atlas2001.wikimedia.org - ayounsi@cumin1001" |
[production] |
| 11:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P51096 and previous config saved to /var/cache/conftool/dbconfig/20230823-111737-ladsgroup.json |
[production] |
| 11:17 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 11:17 |
<ayounsi@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host atlas2001.wikimedia.org |
[production] |
| 11:15 |
<jmm@cumin2002> |
START - Cookbook sre.ldap.roll-restart-reboot-replica rolling reboot on A:ldap-replicas-codfw |
[production] |
| 11:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T343718)', diff saved to https://phabricator.wikimedia.org/P51095 and previous config saved to /var/cache/conftool/dbconfig/20230823-111500-ladsgroup.json |
[production] |
| 11:02 |
<vgutierrez@cumin1001> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp |
[production] |
| 11:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P51094 and previous config saved to /var/cache/conftool/dbconfig/20230823-110231-ladsgroup.json |
[production] |
| 11:01 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and not P{cp2042.*} and A:cp |
[production] |
| 11:00 |
<vgutierrez@cumin1001> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp |
[production] |
| 11:00 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2041.*} and not P{cp2039.*} and A:cp |
[production] |
| 10:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P51093 and previous config saved to /var/cache/conftool/dbconfig/20230823-105954-ladsgroup.json |
[production] |
| 10:54 |
<oblivian@deploy1002> |
helmfile [staging] DONE helmfile.d/services/termbox: apply |
[production] |
| 10:54 |
<oblivian@deploy1002> |
helmfile [staging] START helmfile.d/services/termbox: apply |
[production] |
| 10:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T344589)', diff saved to https://phabricator.wikimedia.org/P51092 and previous config saved to /var/cache/conftool/dbconfig/20230823-104725-ladsgroup.json |
[production] |
| 10:46 |
<vgutierrez@cumin1001> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and not P{cp2042.*} and A:cp |
[production] |
| 10:46 |
<vgutierrez@cumin1001> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2041.*} and not P{cp2039.*} and A:cp |
[production] |
| 10:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P51091 and previous config saved to /var/cache/conftool/dbconfig/20230823-104445-ladsgroup.json |
[production] |
| 10:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T344589)', diff saved to https://phabricator.wikimedia.org/P51090 and previous config saved to /var/cache/conftool/dbconfig/20230823-104308-ladsgroup.json |
[production] |
| 10:40 |
<vgutierrez> |
rolling upgrade to HAProxy 2.6.15 - T344047 |
[production] |
| 10:37 |
<vgutierrez> |
repool cp2039 |
[production] |
| 10:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T343718)', diff saved to https://phabricator.wikimedia.org/P51089 and previous config saved to /var/cache/conftool/dbconfig/20230823-102939-ladsgroup.json |
[production] |
| 10:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P51088 and previous config saved to /var/cache/conftool/dbconfig/20230823-102801-ladsgroup.json |
[production] |
| 10:14 |
<vgutierrez> |
depool cp2039 to run some HAProxy experiments |
[production] |
| 10:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3311', diff saved to https://phabricator.wikimedia.org/P51087 and previous config saved to /var/cache/conftool/dbconfig/20230823-101255-ladsgroup.json |
[production] |
| 10:09 |
<fabfur> |
temporary depool/repool cp4040 for haproxy service restart |
[production] |
| 10:03 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:03 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223 (T343718)', diff saved to https://phabricator.wikimedia.org/P51086 and previous config saved to /var/cache/conftool/dbconfig/20230823-100340-ladsgroup.json |
[production] |
| 09:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3311 (T344589)', diff saved to https://phabricator.wikimedia.org/P51085 and previous config saved to /var/cache/conftool/dbconfig/20230823-095749-ladsgroup.json |
[production] |
| 09:57 |
<klausman@cumin1001> |
START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-codfw |
[production] |
| 09:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2167:3318 (T344589)', diff saved to https://phabricator.wikimedia.org/P51084 and previous config saved to /var/cache/conftool/dbconfig/20230823-095040-ladsgroup.json |
[production] |
| 09:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2167:3311 (T344589)', diff saved to https://phabricator.wikimedia.org/P51083 and previous config saved to /var/cache/conftool/dbconfig/20230823-094916-ladsgroup.json |
[production] |
| 09:49 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance |
[production] |
| 09:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance |
[production] |
| 09:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2166 (T344589)', diff saved to https://phabricator.wikimedia.org/P51082 and previous config saved to /var/cache/conftool/dbconfig/20230823-094851-ladsgroup.json |
[production] |
| 09:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P51081 and previous config saved to /var/cache/conftool/dbconfig/20230823-094834-ladsgroup.json |
[production] |
| 09:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2177 (T343718)', diff saved to https://phabricator.wikimedia.org/P51079 and previous config saved to /var/cache/conftool/dbconfig/20230823-094727-ladsgroup.json |
[production] |
| 09:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 09:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 09:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T343718)', diff saved to https://phabricator.wikimedia.org/P51078 and previous config saved to /var/cache/conftool/dbconfig/20230823-094706-ladsgroup.json |
[production] |
| 09:38 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1010.eqiad.wmnet |
[production] |
| 09:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P51075 and previous config saved to /var/cache/conftool/dbconfig/20230823-093345-ladsgroup.json |
[production] |
| 09:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P51074 and previous config saved to /var/cache/conftool/dbconfig/20230823-093327-ladsgroup.json |
[production] |
| 09:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P51073 and previous config saved to /var/cache/conftool/dbconfig/20230823-093200-ladsgroup.json |
[production] |
| 09:31 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host druid1010.eqiad.wmnet |
[production] |
| 09:31 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1009.eqiad.wmnet |
[production] |