3051-3100 of 10000 results (110ms)
2023-01-10 ยง
09:34 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@9568478]: Fix bug fix in HDFS usage pipeline TEST [airflow-dags@9568478] (duration: 00m 11s) [production]
09:34 <jiji@cumin1001> START - Cookbook sre.dns.netbox [production]
09:34 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@9568478]: Fix bug fix in HDFS usage pipeline TEST [airflow-dags@9568478] [production]
09:25 <XioNoX> repool ulsfo (maintenance cancelled) - T316532 [production]
09:23 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2045.codfw.wmnet [production]
09:22 <taavi> added zabe to wmf-deployment gerrit group T326327 [production]
09:19 <jiji@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc2033.codfw.wmnet [production]
09:18 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2045.codfw.wmnet [production]
09:17 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2032.codfw.wmnet [production]
09:17 <jiji@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:17 <jiji@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2032.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
09:15 <kart_> Done: UTC morning backport window [production]
09:14 <kartik@deploy1002> Finished scap: Backport for [[gerrit:877219|CX: Fix transformation of TranslationUnitDTO to custom array (T326278)]] (duration: 09m 20s) [production]
09:07 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:877219|CX: Fix transformation of TranslationUnitDTO to custom array (T326278)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
09:05 <kartik@deploy1002> Started scap: Backport for [[gerrit:877219|CX: Fix transformation of TranslationUnitDTO to custom array (T326278)]] [production]
08:58 <jiji@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2032.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
08:56 <godog> upgrade thanos to 0.30.1 on thanos-fe1001 - T303154 [production]
08:54 <godog> upgrade thanos to 0.30.1 on prometheus2006 - T303154 [production]
08:49 <kartik@deploy1002> Finished scap: Backport for [[gerrit:877138|CX: Fix usage of categories translation unit as array (T326278)]] (duration: 12m 08s) [production]
08:38 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:877138|CX: Fix usage of categories translation unit as array (T326278)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
08:37 <kartik@deploy1002> Started scap: Backport for [[gerrit:877138|CX: Fix usage of categories translation unit as array (T326278)]] [production]
08:20 <kartik@deploy1002> Finished scap: Backport for [[gerrit:875192|ContentTranslation: Increase MT threshold for publishing in cswiki by 20% (T324721)]] (duration: 17m 21s) [production]
08:09 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
08:09 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
08:08 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:875192|ContentTranslation: Increase MT threshold for publishing in cswiki by 20% (T324721)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
08:03 <kartik@deploy1002> Started scap: Backport for [[gerrit:875192|ContentTranslation: Increase MT threshold for publishing in cswiki by 20% (T324721)]] [production]
08:02 <jiji@cumin1001> START - Cookbook sre.dns.netbox [production]
07:45 <jiji@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc2032.codfw.wmnet [production]
07:37 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2031.codfw.wmnet [production]
07:37 <jiji@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:37 <jiji@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2031.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
07:36 <jiji@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2031.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" [production]
07:33 <jiji@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host mc2044.codfw.wmnet [production]
07:28 <XioNoX> depool ulsfo for network maintenance - T316532 [production]
07:27 <jiji@cumin1001> START - Cookbook sre.dns.netbox [production]
07:22 <jiji@cumin1001> START - Cookbook sre.hosts.decommission for hosts mc2031.codfw.wmnet [production]
07:22 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2044.codfw.wmnet [production]
07:16 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:16 <ayounsi@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: check if dns update is needed after change of rec-dns-lb IPs status - ayounsi@cumin1001" [production]
07:14 <ayounsi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: check if dns update is needed after change of rec-dns-lb IPs status - ayounsi@cumin1001" [production]
07:11 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
07:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
07:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
07:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db1130 T326133', diff saved to https://phabricator.wikimedia.org/P42941 and previous config saved to /var/cache/conftool/dbconfig/20230110-070628-ladsgroup.json [production]
07:03 <XioNoX> remove static routes for legacy dns-rec-lb IPs - T239993 [production]
07:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Promote db1100 to s5 primary and set section read-write T326133', diff saved to https://phabricator.wikimedia.org/P42940 and previous config saved to /var/cache/conftool/dbconfig/20230110-070223-ladsgroup.json [production]
07:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set s5 eqiad as read-only for maintenance - T326133', diff saved to https://phabricator.wikimedia.org/P42939 and previous config saved to /var/cache/conftool/dbconfig/20230110-070152-ladsgroup.json [production]
07:01 <Amir1> Starting s5 eqiad failover from db1130 to db1100 - T326133 [production]
06:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db1100 with weight 0 T326133', diff saved to https://phabricator.wikimedia.org/P42938 and previous config saved to /var/cache/conftool/dbconfig/20230110-062309-ladsgroup.json [production]
06:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s5 T326133 [production]