3151-3200 of 10000 results (126ms)
2024-06-19 ยง
09:40 <fabfur@cumin1002> START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin [production]
09:34 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
09:32 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
09:22 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
09:21 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
09:16 <ayounsi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netbox-dev2003.codfw.wmnet with reason: host reimage [production]
09:15 <claime> Depooling mw2400.codfw.wmnet,mw2403.codfw.wmnet,mw2404.codfw.wmnet,mw2405.codfw.wmnet,mw2408.codfw.wmnet,mw2409.codfw.wmnet for reimage - T351074 [production]
09:13 <ayounsi@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on netbox-dev2003.codfw.wmnet with reason: host reimage [production]
09:11 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
09:10 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe2001.codfw.wmnet with OS bookworm [production]
09:01 <fabfur@cumin1002> END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp5025.*} and A:cp [production]
08:59 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1001.eqiad.wmnet with OS bookworm [production]
08:58 <fabfur@cumin1002> START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp5025.*} and A:cp [production]
08:57 <fabfur@cumin1002> END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp5017.*} and A:cp [production]
08:54 <fabfur@cumin1002> START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp5017.*} and A:cp [production]
08:52 <fabfur> upgrading eqsin cp hosts to haproxy 2.8.10 (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047436) (T367756) [production]
08:51 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage [production]
08:48 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2001.codfw.wmnet with reason: host reimage [production]
08:40 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage [production]
08:39 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 15830 [production]
08:38 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1001.eqiad.wmnet with reason: host reimage [production]
08:35 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'configure' for AS: 15830 [production]
08:31 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host moss-fe2001.codfw.wmnet with OS bookworm [production]
08:30 <mvernon@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-fe2001.codfw.wmnet with OS bookworm [production]
08:30 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on P{ms-fe*} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
08:24 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on P{ms-fe*} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
08:23 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host moss-fe1001.eqiad.wmnet with OS bookworm [production]
08:23 <mvernon@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-fe1001.eqiad.wmnet with OS bookworm [production]
08:18 <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.10 refs T361404 [production]
08:12 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host moss-fe2001.codfw.wmnet with OS bookworm [production]
08:11 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host moss-fe1001.eqiad.wmnet with OS bookworm [production]
08:09 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on P{ms-fe*} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
08:03 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on P{ms-fe*} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
08:01 <ayounsi@cumin1002> START - Cookbook sre.hosts.reimage for host netbox-dev2003.codfw.wmnet with OS bookworm [production]
08:00 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM netbox-dev2003.codfw.wmnet - ayounsi@cumin1002" [production]
07:59 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM netbox-dev2003.codfw.wmnet - ayounsi@cumin1002" [production]
07:59 <ayounsi@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox-dev2003.codfw.wmnet on all recursors [production]
07:59 <ayounsi@cumin1002> START - Cookbook sre.dns.wipe-cache netbox-dev2003.codfw.wmnet on all recursors [production]
07:59 <ayounsi@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:59 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netbox-dev2003.codfw.wmnet - ayounsi@cumin1002" [production]
07:57 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netbox-dev2003.codfw.wmnet - ayounsi@cumin1002" [production]
07:54 <ayounsi@cumin1002> START - Cookbook sre.dns.netbox [production]
07:54 <ayounsi@cumin1002> START - Cookbook sre.ganeti.makevm for new host netbox-dev2003.codfw.wmnet [production]
07:48 <kartik@deploy1002> Finished scap: Backport for [[gerrit:1047382|igwiki: Enable MinT for Wikipedia readers (T363464)]] (duration: 18m 55s) [production]
07:38 <kartik@deploy1002> kartik: Continuing with sync [production]
07:36 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
07:36 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
07:33 <kartik@deploy1002> kartik: Backport for [[gerrit:1047382|igwiki: Enable MinT for Wikipedia readers (T363464)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:29 <kartik@deploy1002> Started scap: Backport for [[gerrit:1047382|igwiki: Enable MinT for Wikipedia readers (T363464)]] [production]
07:22 <kartik@deploy1002> Finished scap: Backport for [[gerrit:1047014|testwiki: Enable MinT for Wikipedia readers MVP on a Igbo Wikipedia (T367852)]] (duration: 20m 12s) [production]