1651-1700 of 10000 results (66ms)
2023-01-24 ยง
14:55 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
14:52 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
14:52 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: sync on main [production]
14:51 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
14:41 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
14:41 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage [production]
14:39 <jiji@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad [production]
14:38 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage [production]
14:36 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:36 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Force update after switch upgrade - volans@cumin1001" [production]
14:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
14:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
14:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
14:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
14:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
14:35 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Force update after switch upgrade - volans@cumin1001" [production]
14:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
14:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
14:34 <jiji@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad [production]
14:33 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
14:29 <jiji@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw [production]
14:29 <effie> switch maps (kartotherian) from eqiad to codfw (attempt #2) [production]
14:28 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
14:28 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
14:25 <TheresNoTime> close UTC afternoon backport window [production]
14:24 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
14:20 <XioNoX> repool ulsfo (maintenance over) [production]
14:20 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bullseye [production]
14:17 <samtar@deploy1002> Finished scap: Backport for [[gerrit:868127|Increase PC writes from parsoid API to 10% (T320534)]] (duration: 07m 41s) [production]
14:14 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
14:11 <samtar@deploy1002> daniel and samtar: Backport for [[gerrit:868127|Increase PC writes from parsoid API to 10% (T320534)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
14:09 <samtar@deploy1002> Started scap: Backport for [[gerrit:868127|Increase PC writes from parsoid API to 10% (T320534)]] [production]
13:50 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
13:44 <XioNoX> reboot ulsfo switches for software upgrade [production]
13:40 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
13:38 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
13:36 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:34 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:30 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ping1002.eqiad.wmnet [production]
13:30 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:30 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ping1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
13:29 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ping1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
13:26 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
13:18 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ping1002.eqiad.wmnet [production]
13:18 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ping2002.codfw.wmnet [production]
13:18 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:18 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ping2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
13:14 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ping2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
13:11 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
13:11 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]