401-450 of 10000 results (108ms)
2025-03-31 ยง
12:23 <cgoubert@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [production]
12:23 <cgoubert@deploy1003> helmfile [codfw] START helmfile.d/services/mw-cron: apply [production]
12:20 <klausman@deploy1003> helmfile [staging] DONE helmfile.d/services/api-gateway: apply [production]
12:20 <klausman@deploy1003> helmfile [staging] START helmfile.d/services/api-gateway: apply [production]
12:18 <dcausse@deploy1003> dcausse: Continuing with sync [production]
12:16 <dcausse@deploy1003> dcausse: Backport for [[gerrit:1132423|Translate: fix elasticsearch cluster setup (T390244)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
12:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4006.ulsfo.wmnet with OS bookworm [production]
12:12 <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1132423|Translate: fix elasticsearch cluster setup (T390244)]] [production]
12:11 <topranks> set "graceful sender" option on cr2-drmrs to darin for JunOS upgrade T364092 [production]
12:08 <claime> Deploying 1131037 mw::periodic_job: Migrate blameStartupRegistry.php - T388540 [production]
12:02 <marostegui@cumin1002> END (ERROR) - Cookbook sre.mysql.clone (exit_code=97) of db1211.eqiad.wmnet onto db1256.eqiad.wmnet [production]
11:56 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4006.ulsfo.wmnet with reason: host reimage [production]
11:53 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4006.ulsfo.wmnet with reason: host reimage [production]
11:51 <Emperor> VACUUM large container dbs on ms-be1066 T377827 [production]
11:49 <mvernon@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1066.eqiad.wmnet with reason: vacuum overlarge container dbs [production]
11:35 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti4006.ulsfo.wmnet with OS bookworm [production]
11:32 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1039.eqiad.wmnet [production]
11:32 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1039.eqiad.wmnet [production]
11:26 <claime> wikikube-worker1039.eqiad.wmnet - powercycling from idrac [production]
11:22 <jmm@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti4006.ulsfo.wmnet [production]
11:22 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet [production]
11:15 <jmm@dns1004> END - running authdns-update [production]
11:13 <jmm@dns1004> START - running authdns-update [production]
11:12 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti4006.ulsfo.wmnet [production]
11:07 <elukey@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on registry1004.eqiad.wmnet with reason: maintenance [production]
11:05 <elukey> remove docker registry nginx cache settings from registry* - T390251 [production]
10:26 <jmm@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti4006.ulsfo.wmnet [production]
10:15 <jmm@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti4006.ulsfo.wmnet with reason: remove from cluster for reimage [production]
10:11 <elukey> restart nginx on registry2005 - T390251 [production]
10:05 <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1132423|Translate: fix elasticsearch cluster setup (T390244)]] [production]
09:58 <fabfur@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp4047.ulsfo.wmnet with reason: HW errors [production]
09:55 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet [production]
09:42 <kevinbazira@deploy1003> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
09:38 <kevinbazira@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
09:31 <kevinbazira@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
09:22 <slyngshede@dns1004> END - running authdns-update [production]
09:20 <slyngshede@dns1004> START - running authdns-update [production]
09:19 <elukey> restart of nginx on registry2004 (by akosiaris) - only instance returning inconsistent responses for a given layer request - T390251 [production]
09:15 <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1132423|Translate: fix elasticsearch cluster setup (T390244)]] [production]
09:13 <elukey> `apt-get clean` on registry200[4,5] to free some space [production]
08:49 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [production]
08:49 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [production]
08:48 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db1211 - Depool db1211.eqiad.wmnet to then clone it to db1256.eqiad.wmnet - marostegui@cumin1002 [production]
08:48 <marostegui@cumin1002> START - Cookbook sre.mysql.depool db1211 - Depool db1211.eqiad.wmnet to then clone it to db1256.eqiad.wmnet - marostegui@cumin1002 [production]
08:48 <marostegui@cumin1002> START - Cookbook sre.mysql.clone of db1211.eqiad.wmnet onto db1256.eqiad.wmnet [production]
08:47 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1003.eqiad.wmnet [production]
08:47 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1003.eqiad.wmnet with OS bookworm [production]
08:38 <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1132423|Translate: fix elasticsearch cluster setup (T390244)]] [production]
08:32 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1003.eqiad.wmnet with reason: host reimage [production]
08:27 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1003.eqiad.wmnet with reason: host reimage [production]