6801-6850 of 10000 results (90ms)
2023-09-01 §
05:31 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet [production]
05:31 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet [production]
05:24 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet [production]
05:17 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet [production]
00:43 <tstarling@deploy1002> Synchronized php-1.41.0-wmf.24/extensions/LoginNotify/includes/Hooks.php: fix production error T345373 (duration: 06m 13s) [production]
00:09 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
00:03 <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Raise LoginNotify minimum log level to info T174200 (duration: 06m 51s) [production]
2023-08-31 §
23:53 <eevans@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host restbase1030.eqiad.wmnet with OS bullseye [production]
23:16 <ryankemper@cumin1001> START - Cookbook sre.wdqs.restart [production]
23:15 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) [production]
23:15 <ryankemper@cumin1001> START - Cookbook sre.wdqs.restart [production]
23:10 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1030.eqiad.wmnet with OS bullseye [production]
22:17 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host doh5002.wikimedia.org with OS bookworm [production]
21:38 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on doh5002.wikimedia.org with reason: host reimage [production]
21:35 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on doh5002.wikimedia.org with reason: host reimage [production]
21:25 <eevans@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host restbase1030.eqiad.wmnet with OS bullseye [production]
21:13 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes2037.codfw.wmnet with OS bullseye [production]
21:08 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts flink-zk2003.codfw.wmnet [production]
21:08 <bking@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:08 <bking@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001" [production]
21:07 <bking@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001" [production]
21:06 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [production]
21:04 <bking@cumin1001> START - Cookbook sre.dns.netbox [production]
21:01 <jhuneidi@deploy1002> Finished scap: Backport for [[gerrit:954049|Use metrics from SiteConfig to restore the Parsoid prefix (T339365)]] (duration: 10m 03s) [production]
21:00 <bking@cumin1001> START - Cookbook sre.hosts.decommission for hosts flink-zk2003.codfw.wmnet [production]
20:57 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host restbase1030.eqiad.wmnet with OS bullseye [production]
20:55 <jhuneidi@deploy1002> arlolra and jhuneidi: Continuing with sync [production]
20:52 <jhuneidi@deploy1002> arlolra and jhuneidi: Backport for [[gerrit:954049|Use metrics from SiteConfig to restore the Parsoid prefix (T339365)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:51 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts flink-zk2001.codfw.wmnet [production]
20:51 <bking@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:51 <bking@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001" [production]
20:51 <jhuneidi@deploy1002> Started scap: Backport for [[gerrit:954049|Use metrics from SiteConfig to restore the Parsoid prefix (T339365)]] [production]
20:50 <bking@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flink-zk2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin1001" [production]
20:47 <bking@cumin1001> START - Cookbook sre.dns.netbox [production]
20:46 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host doh5002.wikimedia.org with OS bookworm [production]
20:45 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for doh6002.wikimedia.org [production]
20:45 <brett@cumin2002> START - Cookbook sre.hosts.remove-downtime for doh6002.wikimedia.org [production]
20:43 <bking@cumin1001> START - Cookbook sre.hosts.decommission for hosts flink-zk2001.codfw.wmnet [production]
20:43 <jhuneidi@deploy1002> Finished scap: Backport for [[gerrit:953660|WatchlistManager: Do not require watchlist rights for clearing talk page notification (T345031)]] (duration: 07m 01s) [production]
20:37 <jhuneidi@deploy1002> jhuneidi and matmarex: Continuing with sync [production]
20:37 <jhuneidi@deploy1002> jhuneidi and matmarex: Backport for [[gerrit:953660|WatchlistManager: Do not require watchlist rights for clearing talk page notification (T345031)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:36 <jhuneidi@deploy1002> Started scap: Backport for [[gerrit:953660|WatchlistManager: Do not require watchlist rights for clearing talk page notification (T345031)]] [production]
20:34 <jhuneidi@deploy1002> Finished scap: Backport for [[gerrit:950046|Undeploy Research Incentive survey on enwiki (T336092)]], [[gerrit:954079|Pre-deploy Campaigns Event Discovery survey (T345158)]] (duration: 14m 19s) [production]
20:32 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host doh6002.wikimedia.org with OS bookworm [production]
20:29 <jhuneidi@deploy1002> jhuneidi and dani: Continuing with sync [production]
20:28 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['restbase1030.eqiad.wmnet'] [production]
20:28 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['restbase1030.eqiad.wmnet'] [production]
20:27 <ryankemper@cumin1001> START - Cookbook sre.wdqs.restart [production]
20:27 <eevans@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['restbase1030.eqiad.wmnet'] [production]
20:27 <eevans@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['restbase1030.eqiad.wmnet'] [production]