651-700 of 10000 results (103ms)
2025-03-10 §
08:09 <marostegui> Failover m1 from db1164 to db1250 - T388024 [production]
08:07 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2232].codfw.wmnet,db[1164,1217,1250].eqiad.wmnet with reason: Primary switchover m1 T388024 [production]
08:05 <awight@deploy2002> Started scap sync-world: Backport for [[gerrit:1125588|Disallow editing modules for non-confirmed/non-autoconfirmed users on the English Wikivoyage (T388301)]] [production]
07:38 <jmm@cumin2002> DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Jdcc-berkman out of all services on: 1284 hosts [production]
07:37 <jmm@cumin2002> DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Jdcc-berkman out of all services on: 961 hosts [production]
2025-03-09 §
10:13 <elukey@puppetserver1001> conftool action : set/weight=5; selector: name=wikikube-worker2.*,dc=codfw,cluster=maps,service=kartotherian-k8s-ssl [production]
10:12 <elukey@puppetserver1001> conftool action : set/weight=5; selector: name=wikikube-worker1.*,dc=eqiad,cluster=maps,service=kartotherian-k8s-ssl [production]
10:12 <elukey@puppetserver1001> conftool action : set/pooled=yes; selector: name=maps2005.codfw.wmnet,dc=codfw,cluster=maps,service=kartotherian-k8s-ssl [production]
10:12 <elukey@puppetserver1001> conftool action : set/pooled=yes; selector: name=maps1005.eqiad.wmnet,dc=eqiad,cluster=maps,service=kartotherian-k8s-ssl [production]
2025-03-08 §
11:33 <moritzm> truncated /var/log/syslog on seaborgium and bounced slapd [production]
00:34 <tzatziki> removing 3 files for legal compliance [production]
2025-03-07 §
22:42 <inflatador> bking@cloudelastic1009 exclude `cloudelastic1010` from master voting T387904 [production]
22:17 <ryankemper> [Cloudelastic] Doing a `/_cluster/reroute?retry_failed=true` of all 3 elastic/opensearch clusters [production]
22:02 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
22:02 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in cloudelastic [production]
22:01 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in cloudelastic [production]
21:36 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudelastic1009.eqiad.wmnet with reason: host reimage [production]
21:32 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudelastic1009.eqiad.wmnet with reason: host reimage [production]
21:13 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
21:12 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
20:58 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
20:49 <bking@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:44 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:42 <bking@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:32 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:32 <bking@cumin2002> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:32 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:17 <bking@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:17 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:17 <bking@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:16 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:16 <bking@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:08 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudelastic1009.eqiad.wmnet'] [production]
20:06 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
19:53 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
19:53 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
19:20 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host cloudelastic1009.eqiad.wmnet with OS bullseye [production]
17:38 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:38 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns additions for eqiad E8/F8 links to new switches - cmooney@cumin1002" [production]
17:38 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns additions for eqiad E8/F8 links to new switches - cmooney@cumin1002" [production]
17:32 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
17:26 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1202.eqiad.wmnet with OS bullseye [production]
17:26 <jclark@cumin1002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
17:18 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:18 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns additions for eqiad E8/F8 links to new switches - cmooney@cumin1002" [production]
17:14 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns additions for eqiad E8/F8 links to new switches - cmooney@cumin1002" [production]
17:08 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
17:06 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
16:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1246', diff saved to https://phabricator.wikimedia.org/P74156 and previous config saved to /var/cache/conftool/dbconfig/20250307-164605-root.json [production]
16:45 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1202.eqiad.wmnet with reason: host reimage [production]