751-800 of 10000 results (14ms)
2025-07-07 §
14:46 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
14:42 <eevans@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
14:41 <vgutierrez> switching lvs6003 to katran - T396561 [production]
14:39 <sukhe> sudo cumin -b1 -s10 'A:wikidough' "run-puppet-agent --enable 'merging CR 1166838'" [production]
14:38 <sukhe> sudo cumin -s1 -b10 'A:wikidough' "run-puppet-agent --enable 'merging CR 1166838'" [production]
14:32 <sukhe> sudo cumin 'A:wikidough' "disable-puppet 'merging CR 1166838'" [production]
14:28 <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
14:26 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
14:25 <eevans@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
14:22 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/api-gateway: sync [production]
14:22 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/api-gateway: sync [production]
14:19 <brouberol> running a rolling-restart of kafka-jumbo - T398826 [analytics]
14:18 <brouberol@cumin2002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-jumbo-eqiad [production]
14:14 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
14:10 <urandom> decommissioning Cassandra/sessionstore-a — T391544 [production]
14:09 <sukhe> sudo cumin -b1 -s10 'A:dnsbox' "run-puppet-agent --enable 'merging CR 1166210'" [production]
14:07 <andrew@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet with OS bookworm [production]
14:05 <jelto@cumin1003> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade Replica to GitLab 18.0 [production]
14:03 <mfossati@deploy1003> Finished deploy [airflow-dags/platform_eng@f79034f]: remove dumps 1.0 sensor from SLIS (duration: 00m 46s) [production]
14:02 <mfossati@deploy1003> Started deploy [airflow-dags/platform_eng@f79034f]: remove dumps 1.0 sensor from SLIS [production]
13:57 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-ctrl1003.eqiad.wmnet [production]
13:54 <jelto@cumin1003> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade Replica to GitLab 18.0 [production]
13:52 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host aux-k8s-ctrl1003.eqiad.wmnet [production]
13:52 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-ctrl2002.codfw.wmnet [production]
13:51 <sukhe> sudo cumin 'A:dnsbox' "disable-puppet 'merging CR 1166210'" [production]
13:49 <andrew@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage [production]
13:48 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host aux-k8s-ctrl2002.codfw.wmnet [production]
13:47 <cgoubert@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet [production]
13:47 <cgoubert@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet [production]
13:46 <cgoubert@cumin1003> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1069.eqiad.wmnet [production]
13:46 <cgoubert@cumin1003> START - Cookbook sre.hosts.remove-downtime for wikikube-worker1069.eqiad.wmnet [production]
13:45 <andrew@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage [production]
13:45 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166828|Revert "Set categorylinks to read new in medium wikis"]] (duration: 07m 59s) [production]
13:45 <claime> homer "cr*eqiad*" commit 'wikikube-worker1069 back to active' [production]
13:42 <cgoubert@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:41 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:41 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:40 <zabe@deploy1003> zabe: Continuing with sync [production]
13:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-ctrl2003.codfw.wmnet [production]
13:39 <cgoubert@cumin1003> START - Cookbook sre.dns.netbox [production]
13:39 <zabe@deploy1003> zabe: Backport for [[gerrit:1166828|Revert "Set categorylinks to read new in medium wikis"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:39 <cgoubert@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:37 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1166828|Revert "Set categorylinks to read new in medium wikis"]] [production]
13:36 <cgoubert@cumin1003> START - Cookbook sre.dns.netbox [production]
13:34 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host aux-k8s-ctrl2003.codfw.wmnet [production]
13:34 <sukhe> sudo cumin -b11 'C:bird' "run-puppet-agent --enable 'merging CR 1166222'": NOOP change [production]
13:31 <zabe@deploy1003> zabe: Continuing with sync [production]
13:30 <zabe@deploy1003> zabe: Backport for [[gerrit:1166816|Set categorylinks to read new in medium wikis (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:28 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1166816|Set categorylinks to read new in medium wikis (T397912)]] [production]
13:28 <sukhe> sudo cumin 'C:bird' "disable-puppet 'merging CR 1166222'" [production]