2601-2650 of 10000 results (82ms)
2023-09-20 ยง
08:59 <godog> restore benthos@webrequest_live running on both centrallog hosts - T346871 [production]
08:57 <klausman> Draining ml-serve1004 for kubelet partition increase (T339231) [production]
08:47 <klausman> Draining ml-serve1003 for kubelet partition increase (T339231) [production]
08:47 <godog> temp bump threads to 15 for benthos@webrequest_live on centrallog2002 - T346871 [production]
08:40 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudservices1005.eqiad.wmnet with OS bullseye [production]
08:40 <aborrero@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudservices1005.eqiad.wmnet with OS bullseye [production]
08:40 <klausman> Draining ml-serve1002 for kubelet partition increase (T339231) [production]
08:36 <godog> stop benthos@webrequest_live.service on centrallog1002 to test redudancy/capacity - T346871 [production]
08:33 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudservices1005.eqiad.wmnet with OS bullseye [production]
08:32 <aborrero@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:31 <aborrero@cumin1001> START - Cookbook sre.dns.netbox [production]
08:31 <aborrero@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudservices1005 [production]
08:31 <aborrero@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host cloudservices1005 [production]
08:30 <aborrero@cumin1001> END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudservices1005 [production]
08:30 <aborrero@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host cloudservices1005 [production]
08:22 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry [production]
08:20 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry [production]
08:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on puppetdb1002.eqiad.wmnet with reason: Disable puppetdb/postgres/nginx on old nodes to ensure nothing hits them anyway [production]
08:16 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on puppetdb1002.eqiad.wmnet with reason: Disable puppetdb/postgres/nginx on old nodes to ensure nothing hits them anyway [production]
08:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on puppetdb2002.codfw.wmnet with reason: Disable puppetdb/postgres/nginx on old nodes to ensure nothing hits them anyway [production]
08:15 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on puppetdb2002.codfw.wmnet with reason: Disable puppetdb/postgres/nginx on old nodes to ensure nothing hits them anyway [production]
08:10 <moritzm> restarting FPM on mw* to pick up libwebp security updates [production]
08:02 <moritzm> installing libwebp security updates on buster [production]
07:42 <slyngshede@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host idm1001.wikimedia.org with OS bookworm [production]
07:41 <taavi@deploy2002> Finished scap: Backport for [[gerrit:959042|Set READ_NEW for Wikitech on OATHAuth multiple devices migration (T242031)]], [[gerrit:959043|Set WRITE_NEW for OATHAuth multiple devices on fishbowls/privates (T242031)]] (duration: 36m 09s) [production]
07:39 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2014.codfw.wmnet [production]
07:34 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2014.codfw.wmnet [production]
07:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-ui1001.eqiad.wmnet [production]
07:29 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-test-ui1001.eqiad.wmnet [production]
07:28 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-web1001.eqiad.wmnet [production]
07:28 <taavi@deploy2002> taavi: Continuing with sync [production]
07:26 <taavi@deploy2002> taavi: Backport for [[gerrit:959042|Set READ_NEW for Wikitech on OATHAuth multiple devices migration (T242031)]], [[gerrit:959043|Set WRITE_NEW for OATHAuth multiple devices on fishbowls/privates (T242031)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental X [production]
07:24 <slyngshede@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idm1001.wikimedia.org with reason: host reimage [production]
07:22 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-web1001.eqiad.wmnet [production]
07:22 <slyngshede@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on idm1001.wikimedia.org with reason: host reimage [production]
07:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1005.eqiad.wmnet [production]
07:14 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-tool1005.eqiad.wmnet [production]
07:10 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet [production]
07:09 <slyngshede@cumin1001> START - Cookbook sre.hosts.reimage for host idm1001.wikimedia.org with OS bookworm [production]
07:06 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet [production]
07:05 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1008.eqiad.wmnet [production]
07:05 <taavi@deploy2002> Started scap: Backport for [[gerrit:959042|Set READ_NEW for Wikitech on OATHAuth multiple devices migration (T242031)]], [[gerrit:959043|Set WRITE_NEW for OATHAuth multiple devices on fishbowls/privates (T242031)]] [production]
07:00 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-tool1008.eqiad.wmnet [production]
06:56 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1009.eqiad.wmnet [production]
06:52 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-tool1009.eqiad.wmnet [production]
06:51 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1010.eqiad.wmnet [production]
06:45 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-tool1010.eqiad.wmnet [production]
06:42 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1011.eqiad.wmnet [production]
06:40 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host an-tool1011.eqiad.wmnet [production]
01:12 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pki1002.eqiad.wmnet with OS bullseye [production]