2251-2300 of 10000 results (49ms)
2022-04-06 ยง
08:28 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host webperf1004.eqiad.wmnet [production]
08:27 <mmandere> pool cp4035 with HAProxy as TLS termination layer - T290005 [production]
08:23 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1001.eqiad.wmnet with reason: host reimage [production]
08:21 <mmandere@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host cp5001.eqsin.wmnet with OS buster [production]
08:20 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4035.ulsfo.wmnet with OS buster [production]
08:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24145 and previous config saved to /var/cache/conftool/dbconfig/20220406-081934-ladsgroup.json [production]
08:18 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1001.eqiad.wmnet with reason: host reimage [production]
08:09 <kharlan@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:747913|Revert "Add zh-hans and zh-hant translation of Module and Module_talk aliases" (T286291 T298308 T165593 T286105)]] (duration: 00m 56s) [production]
08:09 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
08:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
08:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
08:07 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudgw1001.eqiad.wmnet with OS bullseye [production]
08:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
08:03 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host webperf2004.codfw.wmnet [production]
07:55 <kharlan@deploy1002> Synchronized wmf-config: Config: [[gerrit:773204|GrowthExperiments: Add mailing list question for eswiki (T303240 T305015)]] (duration: 00m 56s) [production]
07:47 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4035.ulsfo.wmnet with reason: host reimage [production]
07:44 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:43 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4035.ulsfo.wmnet with reason: host reimage [production]
07:40 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5001.eqsin.wmnet with reason: host reimage [production]
07:38 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
07:38 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host webperf2004.codfw.wmnet [production]
07:36 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5001.eqsin.wmnet with reason: host reimage [production]
07:35 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host webperf2003.codfw.wmnet [production]
07:35 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host bast2002.wikimedia.org [production]
07:30 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1002.wikimedia.org [production]
07:28 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp4035.ulsfo.wmnet with OS buster [production]
07:27 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host urldownloader1002.wikimedia.org [production]
07:24 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
07:24 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
07:23 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
07:23 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
07:23 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
07:23 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
07:21 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
07:20 <mmandere> depool cp4035 for reimage - T290005 [production]
07:19 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:18 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
07:18 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:16 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:12 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp5001.eqsin.wmnet with OS buster [production]
07:12 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:08 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
07:04 <mmandere> depool cp5001 for reimage - T290005 [production]
07:03 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
07:03 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]