351-400 of 10000 results (98ms)
2025-04-01 §
13:35 <fceratto@cumin1002> END (PASS) - Cookbook sre.mysql.zarcillo (exit_code=0) [production]
13:35 <fceratto@cumin1002> START - Cookbook sre.mysql.zarcillo [production]
13:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4008.ulsfo.wmnet with OS bookworm [production]
13:29 <Lucas_WMDE> UTC afternoon backport+config window done [production]
13:28 <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1130299|Remove 'exception-json' logging channel]], [[gerrit:1133124|Disable experiment-related config during active development]] (duration: 18m 04s) [production]
13:27 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2041.codfw.wmnet [production]
13:26 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1040.eqiad.wmnet [production]
13:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2165 (T370903)', diff saved to https://phabricator.wikimedia.org/P74537 and previous config saved to /var/cache/conftool/dbconfig/20250401-132407-ladsgroup.json [production]
13:24 <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
13:21 <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde, cjming, matmarex: Continuing with sync [production]
13:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1160 (T370903)', diff saved to https://phabricator.wikimedia.org/P74536 and previous config saved to /var/cache/conftool/dbconfig/20250401-132059-ladsgroup.json [production]
13:20 <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
13:20 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc1040.eqiad.wmnet [production]
13:20 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc2041.codfw.wmnet [production]
13:18 <moritzm> installing python-cryptography security updates [production]
13:18 <moritzm> installing python-cryptohgraphy security updates [production]
13:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps-test2005.codfw.wmnet with reason: host reimage [production]
13:17 <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde, cjming, matmarex: Backport for [[gerrit:1130299|Remove 'exception-json' logging channel]], [[gerrit:1133124|Disable experiment-related config during active development]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165 (T371742)', diff saved to https://phabricator.wikimedia.org/P74534 and previous config saved to /var/cache/conftool/dbconfig/20250401-131530-ladsgroup.json [production]
13:14 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage [production]
13:13 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps-test2005.codfw.wmnet with reason: host reimage [production]
13:10 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4008.ulsfo.wmnet with reason: host reimage [production]
13:10 <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1130299|Remove 'exception-json' logging channel]], [[gerrit:1133124|Disable experiment-related config during active development]] [production]
13:05 <elukey> restart nginx on registry* to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133112 - debug logs to /var/log/nginx/debug.log - T390251 [production]
13:04 <XioNoX> msw2-eqiad> restart jsd gracefully - T390052 [production]
13:00 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P74533 and previous config saved to /var/cache/conftool/dbconfig/20250401-130023-ladsgroup.json [production]
12:50 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti4008.ulsfo.wmnet with OS bookworm [production]
12:48 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host maps-test2005.codfw.wmnet with OS bookworm [production]
12:47 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps-test2004.codfw.wmnet with OS bookworm [production]
12:47 <fceratto@cumin1002> END (PASS) - Cookbook sre.mysql.zarcillo (exit_code=0) [production]
12:47 <fceratto@cumin1002> START - Cookbook sre.mysql.zarcillo [production]
12:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P74530 and previous config saved to /var/cache/conftool/dbconfig/20250401-124516-ladsgroup.json [production]
12:44 <fceratto@cumin1002> END (PASS) - Cookbook sre.mysql.zarcillo (exit_code=0) [production]
12:44 <fceratto@cumin1002> START - Cookbook sre.mysql.zarcillo [production]
12:43 <fceratto@cumin1002> END (PASS) - Cookbook sre.mysql.zarcillo (exit_code=0) [production]
12:43 <fceratto@cumin1002> START - Cookbook sre.mysql.zarcillo [production]
12:42 <fceratto@cumin1002> END (PASS) - Cookbook sre.mysql.zarcillo (exit_code=0) [production]
12:42 <fceratto@cumin1002> START - Cookbook sre.mysql.zarcillo [production]
12:42 <fceratto@cumin1002> END (FAIL) - Cookbook sre.mysql.zarcillo (exit_code=99) [production]
12:41 <fceratto@cumin1002> START - Cookbook sre.mysql.zarcillo [production]
12:41 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2040.codfw.wmnet [production]
12:41 <fceratto@cumin1002> END (FAIL) - Cookbook sre.mysql.zarcillo (exit_code=99) [production]
12:40 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1039.eqiad.wmnet [production]
12:39 <fceratto@cumin1002> START - Cookbook sre.mysql.zarcillo [production]
12:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti4008.ulsfo.wmnet [production]
12:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet [production]
12:34 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc2040.codfw.wmnet [production]
12:34 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc1039.eqiad.wmnet [production]
12:30 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165 (T371742)', diff saved to https://phabricator.wikimedia.org/P74529 and previous config saved to /var/cache/conftool/dbconfig/20250401-123009-ladsgroup.json [production]
12:29 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet [production]