|
2023-01-23
ยง
|
| 12:06 |
<marostegui> |
dbmaint Reboot db2134 (m3 codfw master) |
[production] |
| 12:05 |
<Emperor> |
removing /usr/local/bin/prometheus-puppet-agent-stats from prometheus crontab on snapshot1014 |
[production] |
| 12:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2107 (T323827)', diff saved to https://phabricator.wikimedia.org/P43241 and previous config saved to /var/cache/conftool/dbconfig/20230123-120012-ladsgroup.json |
[production] |
| 11:58 |
<marostegui> |
dbmaint Reboot db2133 (m2 codfw master) |
[production] |
| 11:57 |
<marostegui> |
dbmaint Reboot db2132 (m1 codfw master) |
[production] |
| 11:57 |
<marostegui> |
Reboot db2132 (m1 codfw master) |
[production] |
| 11:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Maint done', diff saved to https://phabricator.wikimedia.org/P43239 and previous config saved to /var/cache/conftool/dbconfig/20230123-113506-ladsgroup.json |
[production] |
| 11:31 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance |
[production] |
| 11:31 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance |
[production] |
| 11:22 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance |
[production] |
| 11:22 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance |
[production] |
| 11:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depool db2114 T327644', diff saved to https://phabricator.wikimedia.org/P43236 and previous config saved to /var/cache/conftool/dbconfig/20230123-112134-ladsgroup.json |
[production] |
| 11:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2113 (re)pooling @ 75%: Maint done', diff saved to https://phabricator.wikimedia.org/P43235 and previous config saved to /var/cache/conftool/dbconfig/20230123-112001-ladsgroup.json |
[production] |
| 11:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Promote db2129 to s6 primary T327644', diff saved to https://phabricator.wikimedia.org/P43234 and previous config saved to /var/cache/conftool/dbconfig/20230123-111813-ladsgroup.json |
[production] |
| 11:17 |
<Amir1> |
Starting s6 codfw failover from db2114 to db2129 - T327644 |
[production] |
| 11:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2107 (T323827)', diff saved to https://phabricator.wikimedia.org/P43233 and previous config saved to /var/cache/conftool/dbconfig/20230123-111147-ladsgroup.json |
[production] |
| 11:11 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 11:11 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 11:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2113 (re)pooling @ 25%: Maint done', diff saved to https://phabricator.wikimedia.org/P43232 and previous config saved to /var/cache/conftool/dbconfig/20230123-110456-ladsgroup.json |
[production] |
| 10:55 |
<XioNoX> |
update management routers ACLs to add new bast hosts |
[production] |
| 10:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set db2129 with weight 0 T327644', diff saved to https://phabricator.wikimedia.org/P43231 and previous config saved to /var/cache/conftool/dbconfig/20230123-105520-ladsgroup.json |
[production] |
| 10:54 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s6 T327644 |
[production] |
| 10:54 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s6 T327644 |
[production] |
| 10:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2113 (re)pooling @ 10%: Maint done', diff saved to https://phabricator.wikimedia.org/P43230 and previous config saved to /var/cache/conftool/dbconfig/20230123-104951-ladsgroup.json |
[production] |
| 10:48 |
<vgutierrez> |
rolling upgrade to HAProxy 2.4.20 on ulsfo |
[production] |
| 10:40 |
<btullis@deploy1002> |
Finished deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) (duration: 00m 06s) |
[production] |
| 10:40 |
<btullis@deploy1002> |
Started deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) |
[production] |
| 10:40 |
<btullis@deploy1002> |
Finished deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) (duration: 00m 20s) |
[production] |
| 10:40 |
<btullis@deploy1002> |
Started deploy [analytics/superset/deploy@4ba1cb1]: (no justification provided) |
[production] |
| 10:39 |
<btullis@deploy1002> |
Installation of scap version "4.33.1" completed for 1 hosts |
[production] |
| 10:39 |
<btullis@deploy1002> |
Installing scap version "4.33.1" for 1 hosts |
[production] |
| 10:37 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-tool1010.eqiad.wmnet with OS bullseye |
[production] |
| 10:18 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-tool1010.eqiad.wmnet with reason: host reimage |
[production] |
| 10:16 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-tool1010.eqiad.wmnet with reason: host reimage |
[production] |
| 10:07 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:877244|Remove Flow as default in techconductwiki]] (duration: 07m 51s) |
[production] |
| 10:03 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-tool1010.eqiad.wmnet with OS bullseye |
[production] |
| 10:01 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:877244|Remove Flow as default in techconductwiki]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 09:59 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:877244|Remove Flow as default in techconductwiki]] |
[production] |
| 09:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 09:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 09:54 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 09:54 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 09:40 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet |
[production] |
| 09:33 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet |
[production] |
| 09:17 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 09:17 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 09:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 09:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 08:49 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 08:49 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-create ripe-atlas-esams records as the host is back up - volans@cumin1001" |
[production] |