1401-1450 of 10000 results (97ms)
2023-06-08 ยง
13:59 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:58 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:928516|Remove svwiktionary, svwiki and dawiki from legacy encoding (T128156 T128152 T128153)]] (duration: 09m 13s) [production]
13:57 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2014.codfw.wmnet with reason: host reimage [production]
13:54 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs2014.codfw.wmnet with reason: host reimage [production]
13:52 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudswift1002.eqiad.wmnet with reason: host reimage [production]
13:52 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:51 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:51 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:928516|Remove svwiktionary, svwiki and dawiki from legacy encoding (T128156 T128152 T128153)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
13:51 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:50 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:49 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:928516|Remove svwiktionary, svwiki and dawiki from legacy encoding (T128156 T128152 T128153)]] [production]
13:49 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host puppetmaster1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
13:48 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudswift1002.eqiad.wmnet with reason: host reimage [production]
13:44 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:44 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add reverse for new ns-recursor.openstack.codfw1dev.wikimediacloud.org IP. - cmooney@cumin1001" [production]
13:43 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
13:43 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
13:43 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add reverse for new ns-recursor.openstack.codfw1dev.wikimediacloud.org IP. - cmooney@cumin1001" [production]
13:41 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:40 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
13:39 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
13:36 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudswift1002.eqiad.wmnet with OS bullseye [production]
13:35 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host lvs2014.codfw.wmnet with OS bullseye [production]
13:30 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:29 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:06 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
13:06 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
13:05 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
13:05 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
12:57 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
12:57 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
12:36 <cmooney@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: LVS maintenance in eqiad, blocking deploys T322937 (duration: 17m 22s) [production]
12:19 <topranks> De-pooling lvs1017 to move link to lsw1-e1-eqiad to ssw1-e1-eqiad T322937 [production]
12:18 <cmooney@deploy1002> Locking from deployment [ALL REPOSITORIES]: LVS maintenance in eqiad, blocking deploys T322937 [production]
12:12 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:11 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
12:03 <vgutierrez> restore cp4052 HAProxy configuration - T317799 [production]
11:51 <vgutierrez> repooling cp4052 - T317799 [production]
11:40 <vgutierrez> depooling cp4052 for some HAProxy tests - T317799 [production]
11:28 <Amir1> mwscript maintenance/storage/moveToExternal.php --wiki=nlwiki --iconv DB cluster26 (T128154) [production]
11:03 <Amir1> mwscript maintenance/storage/moveToExternal.php --wiki=dawiki --iconv DB cluster27 (T128153) [production]
10:49 <Amir1> mwscript maintenance/storage/moveToExternal.php --wiki=svwiki --iconv DB cluster27 (T128153) [production]
10:22 <jiji@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:21 <jiji@cumin1001> START - Cookbook sre.dns.netbox [production]
09:58 <mfossati@deploy1002> Finished deploy [airflow-dags/platform_eng@bb7526e]: (no justification provided) (duration: 00m 08s) [production]
09:57 <mfossati@deploy1002> Started deploy [airflow-dags/platform_eng@bb7526e]: (no justification provided) [production]
09:40 <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host puppetserver2001.codfw.wmnet with OS bookworm [production]
09:40 <jbond@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jbond@cumin2002" [production]
09:24 <vgutierrez> updated to HAProxy 2.7.9 on cp4052 and cp5032 [production]
09:22 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp5032.eqsin.wmnet,cp4052.ulsfo.wmnet} and A:cp [production]