651-700 of 10000 results (15ms)
2025-07-07 §
19:38 <zabe@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [production]
19:31 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1179.eqiad.wmnet with OS bullseye [production]
19:31 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
19:31 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
19:14 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1179.eqiad.wmnet with reason: host reimage [production]
19:10 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1179.eqiad.wmnet with reason: host reimage [production]
18:59 <bvibber@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]] (duration: 08m 40s) [production]
18:58 <sukhe> sukhe@cp7006:/var/run/confd-template$ sudo rm _etc_haproxy_conf.d_tls.cfg.err [production]
18:55 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye [production]
18:55 <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
18:54 <bvibber@deploy1003> bvibber: Continuing with sync [production]
18:53 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1179.eqiad.wmnet with OS bullseye [production]
18:53 <bvibber@deploy1003> bvibber: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:51 <bvibber@deploy1003> Started scap sync-world: Backport for [[gerrit:1166895|Fix for validation error display in transformed chart data (T398597)]] [production]
18:40 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166890|Revert^2 "Set categorylinks to read new in medium wikis" (T397912)]] (duration: 09m 54s) [production]
18:39 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bullseye [production]
18:35 <zabe@deploy1003> zabe: Continuing with sync [production]
18:32 <zabe@deploy1003> zabe: Backport for [[gerrit:1166890|Revert^2 "Set categorylinks to read new in medium wikis" (T397912)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:31 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1166890|Revert^2 "Set categorylinks to read new in medium wikis" (T397912)]] [production]
18:12 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166886|Apply conditions to correct column (T398823)]] (duration: 11m 14s) [production]
18:10 <urandom> bootstrapping Cassandra/sessionstore1006-a — T391544 [production]
18:09 <sukhe@dns1004> END - running authdns-update [production]
18:09 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) [production]
18:08 <sukhe@dns1004> START - running authdns-update [production]
18:06 <zabe@deploy1003> zabe: Continuing with sync [production]
18:04 <sukhe> [end] rolling upgrade of haproxy on A:dnsbox to 2.6.12-1+deb12u2 [production]
18:04 <sukhe> [emd] rolling upgrade of haproxy on A:dnsbox to 2.6.12-1+deb12u2 [production]
18:03 <zabe@deploy1003> zabe: Backport for [[gerrit:1166886|Apply conditions to correct column (T398823)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:00 <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1166886|Apply conditions to correct column (T398823)]] [production]
17:58 <bking@cumin1002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
17:58 <bking@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
17:58 <ladsgroup@cumin1002> START - Cookbook sre.mysql.sanitarium_restart [production]
17:58 <ladsgroup@cumin1002> END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) [production]
17:57 <ladsgroup@cumin1002> START - Cookbook sre.mysql.sanitarium_restart [production]
17:45 <sukhe> [start] rolling upgrade of haproxy on A:dnsbox to 2.6.12-1+deb12u2 [production]
17:40 <bking@cumin1002> conftool action : set/pooled=true; selector: dnsdisc=search-omega*,name=eqiad [production]
17:40 <bking@cumin1002> conftool action : set/pooled=true; selector: dnsdisc=search-psi*,name=eqiad [production]
17:40 <bking@cumin1002> conftool action : set/pooled=true; selector: dnsdisc=search*,name=eqiad [production]
17:35 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
17:13 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 [tools]
17:12 <bking@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: activate new plugins packages - bking@cumin1002 - T397227 [production]
17:09 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
17:07 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 [tools]
17:07 <bking@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=search-psi*,name=eqiad [production]
17:07 <bking@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=search-omega*,name=eqiad [production]
17:06 <bking@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=search*,name=eqiad [production]
17:05 <eevans@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1006.eqiad.wmnet with reason: host reimage [production]
16:49 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host sessionstore1006.eqiad.wmnet with OS bullseye [production]
16:49 <taavi@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2003-dev.codfw.wmnet [production]
16:49 <eevans@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1006.eqiad.wmnet with OS bullseye [production]