1-50 of 10000 results (57ms)
2024-05-21 ยง
10:46 <hnowlan@cumin1002> START - Cookbook sre.dns.netbox [production]
10:43 <hnowlan@cumin1002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
10:41 <hnowlan@cumin1002> START - Cookbook sre.dns.netbox [production]
10:41 <hnowlan@cumin1002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
10:38 <joal@deploy1002> Finished deploy [analytics/refinery@4d42877]: Deploy of Refinery after reimage of an-launcher1002 [analytics/refinery@4d42877e] (duration: 01m 01s) [production]
10:37 <joal@deploy1002> Started deploy [analytics/refinery@4d42877]: Deploy of Refinery after reimage of an-launcher1002 [analytics/refinery@4d42877e] [production]
10:36 <jmm@cumin2002> END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw [production]
10:34 <hnowlan@cumin1002> START - Cookbook sre.dns.netbox [production]
10:33 <hnowlan@cumin1002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
10:31 <jmm@cumin2002> START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw [production]
10:31 <hnowlan@cumin1002> START - Cookbook sre.dns.netbox [production]
10:24 <aklapper@deploy1002> rebuilt and synchronized wikiversions files: Revert "group0 wikis to 1.43.0-wmf.5" [production]
10:21 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
10:21 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
10:20 <effie> restart memcached on mc2055 [production]
10:18 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db1238.eqiad.wmnet [production]
10:04 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1199.eqiad.wmnet [production]
09:58 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[2331,2361,2391].codfw.wmnet,mw[1372,1429,1436].eqiad.wmnet [production]
09:58 <hnowlan@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:58 <hnowlan@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[2331,2361,2391].codfw.wmnet,mw[1372,1429,1436].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - hnowlan@cumin1002" [production]
09:57 <hnowlan@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[2331,2361,2391].codfw.wmnet,mw[1372,1429,1436].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - hnowlan@cumin1002" [production]
09:57 <moritzm> installing mariadb-10.3 security updates (libs/tools as packaged in Debian, unrelated to wmf-db) [production]
09:56 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db1199.eqiad.wmnet [production]
09:55 <hnowlan@cumin1002> START - Cookbook sre.dns.netbox [production]
09:53 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1190.eqiad.wmnet [production]
09:47 <marostegui@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P62769 and previous config saved to /var/cache/conftool/dbconfig/20240521-094744-root.json [production]
09:41 <aklapper@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.6 refs T361400 [production]
09:36 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db1190.eqiad.wmnet [production]
09:34 <hnowlan@cumin1002> START - Cookbook sre.hosts.decommission for hosts mw[2331,2361,2391].codfw.wmnet,mw[1372,1429,1436].eqiad.wmnet [production]
09:33 <hnowlan> decommissioning 6 appservers in advance of reimaging to k8s control nodes [production]
09:33 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1160.eqiad.wmnet [production]
09:32 <marostegui@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P62768 and previous config saved to /var/cache/conftool/dbconfig/20240521-093238-root.json [production]
09:31 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-launcher1002.eqiad.wmnet with reason: host reimage [production]
09:29 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet1005.eqiad.wmnet with OS bookworm [production]
09:28 <btullis@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on an-launcher1002.eqiad.wmnet with reason: host reimage [production]
09:17 <marostegui@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P62767 and previous config saved to /var/cache/conftool/dbconfig/20240521-091732-root.json [production]
09:16 <btullis@cumin1002> START - Cookbook sre.hosts.reimage for host an-launcher1002.eqiad.wmnet with OS bullseye [production]
09:13 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage [production]
09:10 <taavi@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage [production]
09:09 <tgr|away> UTC morning deploys done [production]
09:06 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db1160.eqiad.wmnet [production]
09:05 <tgr@deploy1002> Finished scap: Backport for [[gerrit:1034173|Temporarily restore $wgCentralAuthDatabase (T348486)]] (duration: 17m 45s) [production]
09:02 <marostegui@cumin1002> dbctl commit (dc=all): 'db1221 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P62766 and previous config saved to /var/cache/conftool/dbconfig/20240521-090224-root.json [production]
09:02 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2219.codfw.wmnet [production]
08:55 <taavi@cumin1002> START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS bookworm [production]
08:51 <tgr@deploy1002> tgr: Continuing with sync [production]
08:51 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db2219.codfw.wmnet [production]
08:50 <tgr@deploy1002> tgr: Backport for [[gerrit:1034173|Temporarily restore $wgCentralAuthDatabase (T348486)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:49 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2210.codfw.wmnet [production]
08:48 <moritzm> installing edk2 security updates [production]