51-100 of 10000 results (14ms)
2025-07-07 §
18:54 <bvibber@deploy1003> bvibber: Continuing with sync [production]
18:53 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye [production]
18:53 <bvibber@deploy1003> bvibber: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:51 <bvibber@deploy1003> Started scap sync-world: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]] [production]
18:40 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166890|Revert^2 "Set categorylinks to read new in medium wikis" (T397912)]] (duration: 09m 54s) [production]
18:39 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye [production]
18:35 <zabe@deploy1003> zabe: Continuing with sync [production]
18:32 <zabe@deploy1003> zabe: Backport for [[gerrit:1166890|Revert^2 "Set categorylinks to read new in medium wikis" (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:31 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1166890|Revert^2 "Set categorylinks to read new in medium wikis" (T397912)]] [production]
18:12 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166886|Apply conditions to correct column (T398823)]] (duration: 11m 14s) [production]
18:10 <urandom> bootstrapping Cassandra/sessionstore1006-a — T391544 [production]
18:09 <sukhe@dns1004> END - running authdns-update [production]
18:09 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) [production]
18:08 <sukhe@dns1004> START - running authdns-update [production]
18:06 <zabe@deploy1003> zabe: Continuing with sync [production]
18:04 <sukhe> [end] rolling upgrade of haproxy on A:dnsbox to 2.6.12-1+deb12u2 [production]
18:04 <sukhe> [emd] rolling upgrade of haproxy on A:dnsbox to 2.6.12-1+deb12u2 [production]
18:03 <zabe@deploy1003> zabe: Backport for [[gerrit:1166886|Apply conditions to correct column (T398823)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:00 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1166886|Apply conditions to correct column (T398823)]] [production]
17:58 <bking@cumin1002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
17:58 <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
17:58 <ladsgroup@cumin1002> START - Cookbook sre.mysql.sanitarium_restart [production]
17:58 <ladsgroup@cumin1002> END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) [production]
17:57 <ladsgroup@cumin1002> START - Cookbook sre.mysql.sanitarium_restart [production]
17:45 <sukhe> [start] rolling upgrade of haproxy on A:dnsbox to 2.6.12-1+deb12u2 [production]
17:40 <bking@cumin1002> conftool action : set/pooled=true; selector: dnsdisc=search-omega*,name=eqiad [production]
17:40 <bking@cumin1002> conftool action : set/pooled=true; selector: dnsdisc=search-psi*,name=eqiad [production]
17:40 <bking@cumin1002> conftool action : set/pooled=true; selector: dnsdisc=search*,name=eqiad [production]
17:35 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
17:13 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 [tools]
17:12 <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
17:09 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
17:07 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 [tools]
17:07 <bking@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=search-psi*,name=eqiad [production]
17:07 <bking@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=search-omega*,name=eqiad [production]
17:06 <bking@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=search*,name=eqiad [production]
17:05 <eevans@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
16:49 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
16:49 <taavi@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2003-dev.codfw.wmnet [production]
16:49 <eevans@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
16:43 <taavi@cumin1003> START - Cookbook sre.hosts.reboot-single for host cloudgw2003-dev.codfw.wmnet [production]
16:39 <dcaro> pushed new ci image docker-registry.svc.toolforge.org/cloud-cicd-py3.11-bookworm-tox:latest [tools]
16:38 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
16:28 <wmbot~lucaswerkmeister@tools-bastion-13> deployed 8ff73093a1 (l10n updates: uk) [tools.wd-image-positions]
16:26 <James_F> jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/ActiveAbstract/ # For T382069 [releng]
16:20 <eevans@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
16:16 <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
16:15 <bking@cumin1002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
16:08 <elukey> kafka preferred-replica-election on kafka1011 to rebalance partition leaders on kafka-jumbo [production]
16:05 <dcaro> clearing images from tools-imagebuilder-2 as it's out of space [tools]