6151-6200 of 10000 results (46ms)
2022-01-17 ยง
11:30 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kafkamon1002.eqiad.wmnet [production]
11:26 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM kafkamon1002.eqiad.wmnet [production]
11:08 <moritzm> switching kubetcd1006 to DRBD-backed storage (required for ganeti update) [production]
11:03 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd1006.eqiad.wmnet with reason: switch to drbd storage [production]
11:03 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd1006.eqiad.wmnet with reason: switch to drbd storage [production]
11:00 <moritzm> systemctl reset-failed ifup@ens5.service on kubetcd1005 T273026 [production]
10:56 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet [production]
10:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove recentchangeslinked group from s3 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18761 and previous config saved to /var/cache/conftool/dbconfig/20220117-104801-marostegui.json [production]
10:47 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet [production]
10:45 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1152.eqiad.wmnet with OS bullseye [production]
10:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T285149)', diff saved to https://phabricator.wikimedia.org/P18760 and previous config saved to /var/cache/conftool/dbconfig/20220117-104459-marostegui.json [production]
10:44 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet [production]
10:42 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1153.eqiad.wmnet with OS bullseye [production]
10:42 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1001.eqiad.wmnet [production]
10:32 <moritzm> switching kubetcd1005 to DRBD-backed storage (required for ganeti update) [production]
10:31 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifeeds: sync on staging [production]
10:31 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd1005.eqiad.wmnet with reason: switch to drbd storage [production]
10:31 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd1005.eqiad.wmnet with reason: switch to drbd storage [production]
10:30 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifeeds: apply on production [production]
10:30 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/wikifeeds: apply on staging [production]
10:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P18759 and previous config saved to /var/cache/conftool/dbconfig/20220117-102954-marostegui.json [production]
10:17 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1152.eqiad.wmnet with OS bullseye [production]
10:15 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1153.eqiad.wmnet with OS bullseye [production]
10:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P18758 and previous config saved to /var/cache/conftool/dbconfig/20220117-101450-marostegui.json [production]
10:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2144.codfw.wmnet with OS bullseye [production]
10:04 <moritzm> switching kubetcd1004 to DRBD-backed storage (required for ganeti update) [production]
10:03 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubetcd1004.eqiad.wmnet with reason: switch to drbd storage [production]
10:03 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on kubetcd1004.eqiad.wmnet with reason: switch to drbd storage [production]
10:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2143.codfw.wmnet with OS bullseye [production]
09:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T285149)', diff saved to https://phabricator.wikimedia.org/P18757 and previous config saved to /var/cache/conftool/dbconfig/20220117-095945-marostegui.json [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1131 (T285149)', diff saved to https://phabricator.wikimedia.org/P18756 and previous config saved to /var/cache/conftool/dbconfig/20220117-095837-marostegui.json [production]
09:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
09:58 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18755 and previous config saved to /var/cache/conftool/dbconfig/20220117-095830-marostegui.json [production]
09:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P18754 and previous config saved to /var/cache/conftool/dbconfig/20220117-094325-marostegui.json [production]
09:30 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2144.codfw.wmnet with OS bullseye [production]
09:30 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2143.codfw.wmnet with OS bullseye [production]
09:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P18753 and previous config saved to /var/cache/conftool/dbconfig/20220117-092820-marostegui.json [production]
09:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1017.eqiad.wmnet with OS bullseye [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18752 and previous config saved to /var/cache/conftool/dbconfig/20220117-091316-marostegui.json [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18751 and previous config saved to /var/cache/conftool/dbconfig/20220117-091308-marostegui.json [production]
09:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
09:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18750 and previous config saved to /var/cache/conftool/dbconfig/20220117-091300-marostegui.json [production]
08:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18749 and previous config saved to /var/cache/conftool/dbconfig/20220117-085756-marostegui.json [production]
08:53 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1017.eqiad.wmnet with OS bullseye [production]
08:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18748 and previous config saved to /var/cache/conftool/dbconfig/20220117-084251-marostegui.json [production]
08:36 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM schema1003.eqiad.wmnet [production]
08:34 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM schema1003.eqiad.wmnet [production]
08:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18747 and previous config saved to /var/cache/conftool/dbconfig/20220117-082746-marostegui.json [production]