501-550 of 10000 results (105ms)
2025-04-01 ยง
12:02 <cjming@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply [production]
12:02 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc2039.codfw.wmnet [production]
11:59 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2204 (T371742)', diff saved to https://phabricator.wikimedia.org/P74528 and previous config saved to /var/cache/conftool/dbconfig/20250401-115935-ladsgroup.json [production]
11:52 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps-test2003.codfw.wmnet with OS bookworm [production]
11:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2204', diff saved to https://phabricator.wikimedia.org/P74527 and previous config saved to /var/cache/conftool/dbconfig/20250401-114428-ladsgroup.json [production]
11:34 <Lucas_WMDE> Deployed patch for T389369 [production]
11:32 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps-test2003.codfw.wmnet with reason: host reimage [production]
11:29 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2204', diff saved to https://phabricator.wikimedia.org/P74526 and previous config saved to /var/cache/conftool/dbconfig/20250401-112921-ladsgroup.json [production]
11:26 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps-test2003.codfw.wmnet with reason: host reimage [production]
11:26 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2038.codfw.wmnet [production]
11:25 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1037.eqiad.wmnet [production]
11:24 <moritzm> installing squid security updates [production]
11:22 <hashar> Restarting Gerrit [production]
11:19 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc2038.codfw.wmnet [production]
11:18 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc1037.eqiad.wmnet [production]
11:16 <jmm@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti4008.ulsfo.wmnet [production]
11:16 <topranks> reboot cr4-ulsfo to upgrade JunOS T364092 [production]
11:15 <hashar> Restarted Gerrit replica on gerrit2002 to raise heap from 32G to 64G | T387223 [production]
11:14 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2204 (T371742)', diff saved to https://phabricator.wikimedia.org/P74525 and previous config saved to /var/cache/conftool/dbconfig/20250401-111415-ladsgroup.json [production]
11:13 <volans@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on sretest1002.eqiad.wmnet with reason: Test [production]
11:12 <moritzm> restarting FPM on phab1004 to pick up security update [production]
11:10 <volans> upgrading spicerack to v10.0.0 on cumin1002 [production]
11:10 <cmooney@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: Upgrade cr4-ulsfo JunOS [production]
11:06 <jmm@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti4008.ulsfo.wmnet with reason: remove from cluster for reimage [production]
11:06 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host maps-test2003.codfw.wmnet with OS bookworm [production]
11:05 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet [production]
11:05 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2006.codfw.wmnet [production]
11:04 <cgoubert@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [production]
11:04 <cgoubert@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-cron: apply [production]
11:04 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wcqs-public [production]
11:04 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps-test2002.codfw.wmnet with OS bookworm [production]
11:02 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wcqs-public [production]
10:58 <ladsgroup@deploy1003> Finished scap sync-world: Backport for [[gerrit:1133087|Bump thumbnail steps to 55% (T360589)]] (duration: 22m 03s) [production]
10:58 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc-gp2006.codfw.wmnet [production]
10:57 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wdqs-all [production]
10:56 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp2005.codfw.wmnet [production]
10:56 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1006.eqiad.wmnet [production]
10:55 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1211.eqiad.wmnet onto db1257.eqiad.wmnet [production]
10:55 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1211 slowly with 10 steps - Pool db1211.eqiad.wmnet in after cloning [production]
10:54 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2204 (T371742)', diff saved to https://phabricator.wikimedia.org/P74523 and previous config saved to /var/cache/conftool/dbconfig/20250401-105425-ladsgroup.json [production]
10:54 <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2204.codfw.wmnet with reason: Maintenance [production]
10:50 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc-gp2005.codfw.wmnet [production]
10:50 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc-gp1006.eqiad.wmnet [production]
10:48 <ladsgroup@deploy1003> ladsgroup: Continuing with sync [production]
10:47 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2165 (T371742)', diff saved to https://phabricator.wikimedia.org/P74522 and previous config saved to /var/cache/conftool/dbconfig/20250401-104659-ladsgroup.json [production]
10:46 <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
10:46 <ladsgroup@deploy1003> ladsgroup: Backport for [[gerrit:1133087|Bump thumbnail steps to 55% (T360589)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
10:45 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wdqs-all [production]
10:44 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wdqs-test [production]
10:43 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wdqs-test [production]