801-850 of 10000 results (66ms)
2022-10-11 ยง
15:30 <TheresNoTime> deployed beta cluster only change, [[gerrit:841547]], for T314294 [production]
15:26 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns4004.wikimedia.org with reason: host reimage [production]
15:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2101.codfw.wmnet with reason: Maintenance [production]
15:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2101.codfw.wmnet with reason: Maintenance [production]
15:25 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4008.ulsfo.wmnet to cluster ulsfo and group 1 [production]
15:23 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns4004.wikimedia.org with reason: host reimage [production]
15:17 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:16 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:10 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:09 <XioNoX> disable cr1-eqiad<->asw2-d-eqiad link for re-cabling - T313463 [production]
15:05 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host dns4004.wikimedia.org with OS buster [production]
15:04 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti4008.ulsfo.wmnet to cluster ulsfo and group 1 [production]
15:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:58 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
14:56 <XioNoX> re-enable cr1-eqiad<->asw2-c-eqiad link after optic replacement [production]
14:51 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti4008.ulsfo.wmnet to cluster ulsfo and group 1 [production]
14:51 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti4008.ulsfo.wmnet to cluster ulsfo and group 1 [production]
14:50 <XioNoX> disable cr1-eqiad<->asw2-c-eqiad link for optic replacement [production]
14:49 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet [production]
14:27 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:26 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:26 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet [production]
14:25 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
14:15 <sukhe> completed homer run for Gerrit 841501 [production]
14:14 <sukhe> homer "cr*-ulsfo*" commit "Gerrit 841501: sites.yaml: decom dns4002" [production]
14:05 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:03 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:01 <hoo@deploy1002> Finished scap: Backport for [[gerrit:841164|updateQueryServiceLag: Add lb(-pool) options for forward compatibility (T315423 T238751)]] (duration: 04m 57s) [production]
14:00 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:57 <hoo@deploy1002> hoo and hoo: Backport for [[gerrit:841164|updateQueryServiceLag: Add lb(-pool) options for forward compatibility (T315423 T238751)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
13:56 <hoo@deploy1002> Started scap: Backport for [[gerrit:841164|updateQueryServiceLag: Add lb(-pool) options for forward compatibility (T315423 T238751)]] [production]
13:50 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lvs4008.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
13:19 <jgiannelos@deploy1002> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
13:18 <jgiannelos@deploy1002> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
13:18 <jgiannelos@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
13:17 <jgiannelos@deploy1002> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
13:17 <jgiannelos@deploy1002> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
13:17 <jgiannelos@deploy1002> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
13:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:14 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.40.0-wmf.4/extensions/Wikistories/extension.json: Backport: [[gerrit:840178|Make discovery mode config default to 'off' (T314582)]] (duration: 03m 48s) [production]
13:14 <volans@cumin2002> START - Cookbook sre.hosts.provision for host lvs4008.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
13:13 <volans@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host lvs4008.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
13:12 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:02 <jgiannelos@deploy1002> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
13:01 <jgiannelos@deploy1002> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
13:01 <jgiannelos@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]