|
2023-03-29
ยง
|
| 21:24 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host gerrit1003.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 21:24 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host gerrit1003.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 21:23 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host gerrit1003.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 21:15 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs4010.ulsfo.wmnet |
[production] |
| 21:15 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for lvs4010.ulsfo.wmnet |
[production] |
| 20:52 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS bullseye |
[production] |
| 20:29 |
<taavi@deploy2002> |
Finished scap: Backport for [[gerrit:893839|Add per-action component-level profiling in statsd using excimer (T225968)]] (duration: 11m 52s) |
[production] |
| 20:28 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage |
[production] |
| 20:28 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1072.eqiad.wmnet'] |
[production] |
| 20:28 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1072.eqiad.wmnet'] |
[production] |
| 20:27 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-be1072.eqiad.wmnet'] |
[production] |
| 20:26 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1072.eqiad.wmnet'] |
[production] |
| 20:26 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage |
[production] |
| 20:25 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1072.eqiad.wmnet'] |
[production] |
| 20:25 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1072.eqiad.wmnet'] |
[production] |
| 20:24 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1073'] |
[production] |
| 20:24 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1073'] |
[production] |
| 20:18 |
<taavi@deploy2002> |
aaron and taavi: Backport for [[gerrit:893839|Add per-action component-level profiling in statsd using excimer (T225968)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:17 |
<taavi@deploy2002> |
Started scap: Backport for [[gerrit:893839|Add per-action component-level profiling in statsd using excimer (T225968)]] |
[production] |
| 20:15 |
<taavi@deploy2002> |
Finished scap: Backport for [[gerrit:903835|Update "United States" static page to facilitate synthetic testing of T331681 (T331681)]] (duration: 09m 45s) |
[production] |
| 20:10 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS bullseye |
[production] |
| 20:10 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1211.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:07 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1075.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:07 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host db1210.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:07 |
<taavi@deploy2002> |
nray and taavi: Backport for [[gerrit:903835|Update "United States" static page to facilitate synthetic testing of T331681 (T331681)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:06 |
<volans@cumin1001> |
START - Cookbook sre.hosts.provision for host db1211.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:06 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:05 |
<taavi@deploy2002> |
Started scap: Backport for [[gerrit:903835|Update "United States" static page to facilitate synthetic testing of T331681 (T331681)]] |
[production] |
| 20:05 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.provision for host ms-be1075.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:04 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1074.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:03 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.provision for host ms-be1074.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:56 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:50 |
<btullis@cumin1001> |
END (FAIL) - Cookbook sre.wikireplicas.add-wiki (exit_code=99) |
[production] |
| 19:50 |
<btullis@cumin1001> |
START - Cookbook sre.wikireplicas.add-wiki |
[production] |
| 19:48 |
<volans@cumin1001> |
START - Cookbook sre.hosts.provision for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:20 |
<sukhe> |
force puppet agent run on A:lvs to additionally confirm nothing broke |
[production] |
| 19:20 |
<sukhe> |
[enable] puppet on A:lvs to roll out pybal prometheus-client change |
[production] |
| 19:14 |
<sukhe> |
disable puppet on A:lvs to roll out pybal prometheus-client change |
[production] |
| 18:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:55 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 18:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depool db1138 T333480', diff saved to https://phabricator.wikimedia.org/P45981 and previous config saved to /var/cache/conftool/dbconfig/20230329-185431-ladsgroup.json |
[production] |
| 18:54 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 18:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Promote db1160 to s4 primary T333480', diff saved to https://phabricator.wikimedia.org/P45980 and previous config saved to /var/cache/conftool/dbconfig/20230329-185125-ladsgroup.json |
[production] |
| 18:50 |
<Amir1> |
Starting s4 eqiad failover from db1138 to db1160 - T333480 |
[production] |
| 18:48 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host gerrit1003.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 18:48 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 18:48 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 18:47 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 18:46 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1209.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |