4951-5000 of 10000 results (83ms)
2022-06-14 ยง
18:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:00 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1053.eqiad.wmnet [production]
17:57 <brennen@deploy1002> Pruned MediaWiki: 1.39.0-wmf.14 (duration: 01m 53s) [production]
17:56 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:55 <brennen@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.16 (duration: 32m 52s) [production]
17:31 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:30 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:25 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1053.eqiad.wmnet [production]
17:24 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:23 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:22 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:22 <brennen@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.16 [production]
17:12 <brennen> train 1.39.0-wmf.16 (T308069): train is blocked - will sync to testwikis and hold there for resolution of T310532 [production]
16:23 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2053.codfw.wmnet with OS bullseye [production]
16:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
16:20 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
16:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
16:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
16:18 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1052.eqiad.wmnet [production]
16:12 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1052.eqiad.wmnet [production]
16:12 <jnuche@deploy1002> Installation of scap version "4.9.2" completed for 557 hosts [production]
16:11 <jnuche@deploy1002> Installing scap version "4.9.2" for 557 hosts [production]
16:05 <jnuche@deploy1002> Installing scap version "4.9.2" for 557 hosts [production]
16:01 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2053.codfw.wmnet with reason: host reimage [production]
15:58 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2053.codfw.wmnet with reason: host reimage [production]
15:34 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]
15:21 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host elastic2053.codfw.wmnet [production]
15:19 <pt1979@cumin2002> START - Cookbook sre.hosts.dhcp for host elastic2053.codfw.wmnet [production]
15:09 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1051.eqiad.wmnet [production]
14:54 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:53 <moritzm> failover ganeti master in ulsfo to ganeti4003 [production]
14:53 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:53 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
14:49 <urbanecm@deploy1002> Synchronized wmf-config/throttle.php: 596058b5e4d906d40e620fe5b01f37c484f5a8c1: Add new throttle rule + remove expired one (T310625) (duration: 03m 38s) [production]
14:40 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: reboots [production]
14:40 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: reboots [production]
14:33 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1051.eqiad.wmnet [production]
14:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: reboots [production]
14:33 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: reboots [production]
14:29 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4003.ulsfo.wmnet [production]
14:22 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti4003.ulsfo.wmnet [production]
14:20 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2012.codfw.wmnet with OS buster [production]
14:18 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2010.codfw.wmnet with OS buster [production]
14:16 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1002.eqiad.wmnet [production]
14:16 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2009.codfw.wmnet with OS buster [production]
14:15 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host failoid1002.eqiad.wmnet [production]