1-50 of 10000 results (3ms)
2026-09-08 ยง
17:15 <swfrench@deploy1003> Finished scap sync-world: Noop deployment to test mediawiki_runtime_image cleanup (duration: 04m 18s) [production]
17:11 <Amir1> dropping links tables from db1247 (s4 replica) - (T437278) [production]
17:10 <swfrench@deploy1003> Started scap sync-world: Noop deployment to test mediawiki_runtime_image cleanup [production]
17:04 <larssandergreen> civicrm upgraded from e66e5440 to 13b49ffe [fundraising]
16:51 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1337946|Use local database for category table in SpecialWantedCategories (T437286)]], [[gerrit:1337947|Use local database for category table in SpecialWantedCategories (T437286)]] (duration: 10m 19s) [production]
16:46 <zabe@deploy1003> zabe: Continuing with deployment [production]
16:45 <zabe@deploy1003> zabe: Backport for [[gerrit:1337946|Use local database for category table in SpecialWantedCategories (T437286)]], [[gerrit:1337947|Use local database for category table in SpecialWantedCategories (T437286)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
16:41 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1337946|Use local database for category table in SpecialWantedCategories (T437286)]], [[gerrit:1337947|Use local database for category table in SpecialWantedCategories (T437286)]] [production]
16:38 <larssandergreen> tools upgraded from 14ec70fa to c7be1099 [fundraising]
16:29 <jhancock@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage [production]
16:25 <jhancock@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage [production]
16:25 <btullis@puppetserver1001> conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2005.codfw.wmnet [production]
16:25 <btullis@puppetserver1001> conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2004.codfw.wmnet [production]
16:24 <btullis@puppetserver1001> conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2005.codfw.wmnet [production]
16:24 <btullis@puppetserver1001> conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2004.codfw.wmnet [production]
16:21 <thcipriani> hard reboot deployment-prep:deployment-cache-text08 horizon logs showing OOMs [releng]
16:12 <jhancock@cumin2003> START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie [production]
16:08 <jhancock@cumin2003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
16:05 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [production]
16:05 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [production]
16:04 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd2003.codfw.wmnet [production]
15:58 <jhancock@cumin2003> START - Cookbook sre.hosts.provision for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:55 <btullis@cumin1003> START - Cookbook sre.hosts.reboot-single for host cephosd2003.codfw.wmnet [production]
15:54 <jhancock@cumin2003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:44 <jhancock@cumin2003> START - Cookbook sre.hosts.provision for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:42 <hashar> Updating Quibble jobs to 1.21.0 # T429715 T303270 T437134 T300727 [releng]
15:29 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd2002.codfw.wmnet [production]
15:19 <btullis@cumin1003> START - Cookbook sre.hosts.reboot-single for host cephosd2002.codfw.wmnet [production]
15:06 <hashar> Tag Quibble @ b692707e5213abf5e16894b1fb21cea41cd72c9e # T429715 T303270 T437134 T300727 [releng]
14:55 <logmsgbot> dreamyjazz Deployed security patch for T436429 [production]
14:46 <logmsgbot> dreamyjazz Deployed security patch for T436429 [production]
14:45 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd2001.codfw.wmnet [production]
14:44 <topranks> shutdown et-1/1/5 on cr1-codfw to shift traffic off ssw1-a1-codfw [production]
14:43 <cmooney@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: upgrade ssw1-a1-eqiad [production]
14:40 <komla@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T436614) [tools]
14:34 <btullis@cumin1003> START - Cookbook sre.hosts.reboot-single for host cephosd2001.codfw.wmnet [production]
14:33 <btullis@cumin1003> END (ERROR) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=97) rolling reboot on A:cephosd-codfw [production]
14:30 <btullis@cumin1003> START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw [production]
14:28 <klausman@cumin1004> END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad [production]
14:28 <klausman@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet [production]
14:28 <klausman@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet [production]
14:22 <klausman@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet [production]
14:18 <komla@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T436614) [tools]
14:13 <komla@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers (T436614) [tools]
14:13 <komla@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T436614) [tools]
14:13 <urbanecm@deploy1003> mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan 1788220800 --verbose # T437158 [production]
14:12 <klausman@cumin1004> START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet [production]
14:12 <klausman@cumin1004> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet [production]
14:12 <klausman@cumin1004> START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet [production]
14:03 <topranks> drain traffic from ssw1-a1-codfw before JunOS upgrade T426197 [production]