2401-2450 of 10000 results (74ms)
2023-01-10 ยง
12:05 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
12:02 <jiji@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2034.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
11:59 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [production]
11:58 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
11:57 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
11:57 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
11:56 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
11:53 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
11:52 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
11:48 <jiji@cumin1001> START - Cookbook sre.dns.netbox [production]
11:35 <btullis@cumin1001> START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes [production]
11:33 <btullis@cumin1001> END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid public cluster: Reboot Druid nodes [production]
11:06 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2047.codfw.wmnet [production]
11:00 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2047.codfw.wmnet [production]
11:00 <jiji@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc2034.codfw.wmnet [production]
10:31 <godog> upgrade thanos to 0.30.1 on thanos-fe2* - T303154 [production]
10:24 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host cephosd1002.eqiad.wmnet with OS bullseye [production]
10:23 <btullis@cumin1001> START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes [production]
10:21 <claime> Starting rolling reboot of eqiad jobrunners [production]
10:21 <cgoubert@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
10:18 <btullis@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cephosd1002.eqiad.wmnet with OS bullseye [production]
10:14 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2046.codfw.wmnet [production]
10:14 <claime> repooled parse1002.eqiad.wmnet - T326119 [production]
10:13 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1002.eqiad.wmnet [production]
10:13 <cgoubert@cumin1001> START - Cookbook sre.hosts.remove-downtime for parse1002.eqiad.wmnet [production]
10:07 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2046.codfw.wmnet [production]
10:06 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2033.codfw.wmnet [production]
10:06 <jiji@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:06 <jiji@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2033.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
10:02 <jiji@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2033.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
09:59 <cgoubert@cumin1001> conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1002.eqiad.wmnet [production]
09:55 <godog> upgrade thanos to 0.30.1 on prometheus hosts - T303154 [production]
09:53 <moritzm> installing systemd bugfix updates from Bullseye point release [production]
09:45 <aqu@deploy1002> Finished deploy [airflow-dags/analytics@9568478]: Fix bug fix in HDFS usage pipeline [airflow-dags@9568478] (duration: 00m 13s) [production]
09:45 <aqu@deploy1002> Started deploy [airflow-dags/analytics@9568478]: Fix bug fix in HDFS usage pipeline [airflow-dags@9568478] [production]
09:43 <godog> upgrade thanos to 0.30.1 on thanos-fe100[2-3] - T303154 [production]
09:34 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@9568478]: Fix bug fix in HDFS usage pipeline TEST [airflow-dags@9568478] (duration: 00m 11s) [production]
09:34 <jiji@cumin1001> START - Cookbook sre.dns.netbox [production]
09:34 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@9568478]: Fix bug fix in HDFS usage pipeline TEST [airflow-dags@9568478] [production]
09:25 <XioNoX> repool ulsfo (maintenance cancelled) - T316532 [production]
09:23 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2045.codfw.wmnet [production]
09:22 <taavi> added zabe to wmf-deployment gerrit group T326327 [production]
09:19 <jiji@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc2033.codfw.wmnet [production]
09:18 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2045.codfw.wmnet [production]
09:17 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2032.codfw.wmnet [production]
09:17 <jiji@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:17 <jiji@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2032.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
09:15 <kart_> Done: UTC morning backport window [production]
09:14 <kartik@deploy1002> Finished scap: Backport for [[gerrit:877219|CX: Fix transformation of TranslationUnitDTO to custom array (T326278)]] (duration: 09m 20s) [production]
09:07 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:877219|CX: Fix transformation of TranslationUnitDTO to custom array (T326278)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]