701-750 of 10000 results (78ms)
2024-02-14 ยง
10:58 <akosiaris@deploy2002> helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply [production]
10:58 <aokoth@cumin1002> START - Cookbook sre.ganeti.reboot-vm for VM vrts1002.eqiad.wmnet [production]
10:58 <akosiaris@deploy2002> helmfile [staging] START helmfile.d/services/eventstreams-internal: apply [production]
10:58 <akosiaris@deploy2002> helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply [production]
10:57 <akosiaris@deploy2002> helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply [production]
10:57 <akosiaris@deploy2002> helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply [production]
10:56 <akosiaris@deploy2002> helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply [production]
10:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P56747 and previous config saved to /var/cache/conftool/dbconfig/20240214-105530-ladsgroup.json [production]
10:52 <ayounsi@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2005.codfw.wmnet with OS bookworm [production]
10:48 <akosiaris@deploy2002> helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [production]
10:48 <akosiaris@deploy2002> helmfile [codfw] START helmfile.d/services/eventstreams: apply [production]
10:48 <jelto> import prometheus-etherpad-exporter 0.7 to bookworm-wikimedia on apt hosts - T316421 [production]
10:46 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=maps2005.codfw.wmnet [production]
10:45 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=maps2005.codfw.wmnet [production]
10:41 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host puppetserver2003.codfw.wmnet with OS bookworm [production]
10:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T352010)', diff saved to https://phabricator.wikimedia.org/P56746 and previous config saved to /var/cache/conftool/dbconfig/20240214-104024-ladsgroup.json [production]
10:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1166 (T352010)', diff saved to https://phabricator.wikimedia.org/P56745 and previous config saved to /var/cache/conftool/dbconfig/20240214-103810-ladsgroup.json [production]
10:38 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
10:37 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
10:37 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
10:37 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
10:37 <slyngs> Deploying new PKI checks to alertmanager [production]
10:33 <akosiaris@deploy2002> helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply [production]
10:33 <akosiaris@deploy2002> helmfile [eqiad] START helmfile.d/services/eventstreams: apply [production]
10:31 <jmm@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host puppetserver2003.codfw.wmnet with OS bookworm [production]
10:28 <akosiaris@deploy2002> helmfile [staging] DONE helmfile.d/services/eventstreams: apply [production]
10:28 <akosiaris@deploy2002> helmfile [staging] START helmfile.d/services/eventstreams: apply [production]
10:19 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host puppetserver2003.codfw.wmnet with OS bookworm [production]
10:18 <godog> powercycle titan1001 [production]
10:02 <ayounsi@cumin1002> START - Cookbook sre.hosts.reimage for host sretest2005.codfw.wmnet with OS bookworm [production]
09:55 <moritzm> installing Linux 5.10.209 on Bullseye hosts [production]
09:49 <moritzm> imported openssl11 1.1.1w-0+deb11u1+wmf1 to component/haproxy26 T352744 [production]
09:42 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet [production]
09:38 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet [production]
09:08 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet [production]
09:05 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet [production]
08:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1221 (T352010)', diff saved to https://phabricator.wikimedia.org/P56744 and previous config saved to /var/cache/conftool/dbconfig/20240214-084146-ladsgroup.json [production]
08:41 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
08:41 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
08:41 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance [production]
08:41 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance [production]
08:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1199 (T352010)', diff saved to https://phabricator.wikimedia.org/P56743 and previous config saved to /var/cache/conftool/dbconfig/20240214-084104-ladsgroup.json [production]
08:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P56742 and previous config saved to /var/cache/conftool/dbconfig/20240214-082558-ladsgroup.json [production]
08:20 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1001.wikimedia.org [production]
08:12 <taavi> restart apache2 on lists1001 to remove traces of old, soon-to-expire TLS certificate [production]
08:11 <taavi@cumin1002> START - Cookbook sre.hosts.reboot-single for host clouddumps1001.wikimedia.org [production]
08:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P56741 and previous config saved to /var/cache/conftool/dbconfig/20240214-081051-ladsgroup.json [production]
07:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1199 (T352010)', diff saved to https://phabricator.wikimedia.org/P56740 and previous config saved to /var/cache/conftool/dbconfig/20240214-075545-ladsgroup.json [production]
07:51 <stran@deploy2002> helmfile [codfw] DONE helmfile.d/services/ipoid: apply [production]
07:50 <stran@deploy2002> helmfile [codfw] START helmfile.d/services/ipoid: apply [production]