|
2023-05-08
ยง
|
| 20:32 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:15:00 on miscweb2003.codfw.wmnet with reason: reboot |
[production] |
| 20:32 |
<herron@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kafka-logging1003.eqiad.wmnet |
[production] |
| 20:31 |
<taavi@deploy1002> |
jdlrobson and taavi: Backport for [[gerrit:917395|Deploy fixed width indicator to wikis (T335307)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P47941 and previous config saved to /var/cache/conftool/dbconfig/20230508-203043-ladsgroup.json |
[production] |
| 20:30 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:917395|Deploy fixed width indicator to wikis (T335307)]] |
[production] |
| 20:29 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:917162|Ensure page load popupNotification is closed when the toggle button is clicked (T335153)]] (duration: 07m 58s) |
[production] |
| 20:25 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2007.codfw.wmnet |
[production] |
| 20:24 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on vrts1001.eqiad.wmnet with reason: host reimage |
[production] |
| 20:23 |
<taavi@deploy1002> |
jdlrobson and taavi: Backport for [[gerrit:917162|Ensure page load popupNotification is closed when the toggle button is clicked (T335153)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:21 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:917162|Ensure page load popupNotification is closed when the toggle button is clicked (T335153)]] |
[production] |
| 20:21 |
<aokoth@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on vrts1001.eqiad.wmnet with reason: host reimage |
[production] |
| 20:20 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:917160|Update a/b test code for visual enhancements a/b test (T333715)]], [[gerrit:916903|Enable DiscussionTools visual enhancements a/b test (T302358)]] (duration: 11m 54s) |
[production] |
| 20:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T335845)', diff saved to https://phabricator.wikimedia.org/P47939 and previous config saved to /var/cache/conftool/dbconfig/20230508-201537-ladsgroup.json |
[production] |
| 20:11 |
<aokoth@cumin1001> |
START - Cookbook sre.ganeti.reimage for host vrts1001.eqiad.wmnet with OS bullseye |
[production] |
| 20:09 |
<taavi@deploy1002> |
kemayo and taavi: Backport for [[gerrit:917160|Update a/b test code for visual enhancements a/b test (T333715)]], [[gerrit:916903|Enable DiscussionTools visual enhancements a/b test (T302358)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1180 (T335845)', diff saved to https://phabricator.wikimedia.org/P47938 and previous config saved to /var/cache/conftool/dbconfig/20230508-200825-ladsgroup.json |
[production] |
| 20:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:08 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:917160|Update a/b test code for visual enhancements a/b test (T333715)]], [[gerrit:916903|Enable DiscussionTools visual enhancements a/b test (T302358)]] |
[production] |
| 20:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1173 (T335845)', diff saved to https://phabricator.wikimedia.org/P47937 and previous config saved to /var/cache/conftool/dbconfig/20230508-200802-ladsgroup.json |
[production] |
| 20:05 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host vrts1001.eqiad.wmnet |
[production] |
| 20:05 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM vrts1001.eqiad.wmnet - aokoth@cumin1001" |
[production] |
| 20:04 |
<aokoth@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM vrts1001.eqiad.wmnet - aokoth@cumin1001" |
[production] |
| 20:04 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) vrts1001.eqiad.wmnet on all recursors |
[production] |
| 20:04 |
<aokoth@cumin1001> |
START - Cookbook sre.dns.wipe-cache vrts1001.eqiad.wmnet on all recursors |
[production] |
| 20:04 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:04 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM vrts1001.eqiad.wmnet - aokoth@cumin1001" |
[production] |
| 20:03 |
<aokoth@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM vrts1001.eqiad.wmnet - aokoth@cumin1001" |
[production] |
| 19:59 |
<aokoth@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:59 |
<aokoth@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host vrts1001.eqiad.wmnet |
[production] |
| 19:54 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-logging1005.eqiad.wmnet |
[production] |
| 19:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P47936 and previous config saved to /var/cache/conftool/dbconfig/20230508-195256-ladsgroup.json |
[production] |
| 19:52 |
<bking@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host wdqs2007.codfw.wmnet |
[production] |
| 19:51 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2004.codfw.wmnet |
[production] |
| 19:46 |
<herron@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kafka-logging1005.eqiad.wmnet |
[production] |
| 19:45 |
<bking@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host wdqs2004.codfw.wmnet |
[production] |
| 19:45 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-logging2004.codfw.wmnet |
[production] |
| 19:41 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2006.codfw.wmnet |
[production] |
| 19:39 |
<herron@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kafka-logging2004.codfw.wmnet |
[production] |
| 19:38 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-logging2005.codfw.wmnet |
[production] |
| 19:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P47935 and previous config saved to /var/cache/conftool/dbconfig/20230508-193750-ladsgroup.json |
[production] |
| 19:34 |
<bking@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host wdqs2006.codfw.wmnet |
[production] |
| 19:32 |
<herron@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kafka-logging2005.codfw.wmnet |
[production] |
| 19:31 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-logging2003.codfw.wmnet |
[production] |
| 19:24 |
<herron@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kafka-logging2003.codfw.wmnet |
[production] |
| 19:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1173 (T335845)', diff saved to https://phabricator.wikimedia.org/P47934 and previous config saved to /var/cache/conftool/dbconfig/20230508-192243-ladsgroup.json |
[production] |
| 19:20 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on wdqs2006.codfw.wmnet with reason: rebooting to help with lag |
[production] |
| 19:20 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on wdqs2006.codfw.wmnet with reason: rebooting to help with lag |
[production] |
| 19:20 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs2006.codfw.wmnet |
[production] |
| 19:20 |
<bking@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for wdqs2006.codfw.wmnet |
[production] |