601-650 of 10000 results (65ms)
2022-10-27 ยง
08:17 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1009.eqiad.wmnet to cluster eqiad and group C [production]
08:17 <marostegui@deploy1002> Started scap: Backport for [[gerrit:850029|ProductionServices.php: Promote pc1014 to pc3 master]] [production]
08:16 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1009.eqiad.wmnet to cluster eqiad and group C [production]
08:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
08:15 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
08:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318950)', diff saved to https://phabricator.wikimedia.org/P36667 and previous config saved to /var/cache/conftool/dbconfig/20221027-081534-ladsgroup.json [production]
08:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
08:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
08:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
08:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1024 (T321312)', diff saved to https://phabricator.wikimedia.org/P36666 and previous config saved to /var/cache/conftool/dbconfig/20221027-081339-ladsgroup.json [production]
08:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
08:11 <jnuche@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.7 refs T320512 [production]
08:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1166 (T318950)', diff saved to https://phabricator.wikimedia.org/P36665 and previous config saved to /var/cache/conftool/dbconfig/20221027-081113-ladsgroup.json [production]
08:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
08:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
08:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318950)', diff saved to https://phabricator.wikimedia.org/P36664 and previous config saved to /var/cache/conftool/dbconfig/20221027-081103-ladsgroup.json [production]
08:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es2023', diff saved to https://phabricator.wikimedia.org/P36663 and previous config saved to /var/cache/conftool/dbconfig/20221027-080625-ladsgroup.json [production]
08:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P36662 and previous config saved to /var/cache/conftool/dbconfig/20221027-080027-ladsgroup.json [production]
07:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P36661 and previous config saved to /var/cache/conftool/dbconfig/20221027-075556-ladsgroup.json [production]
07:55 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd1005.eqiad.wmnet to plain [production]
07:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es1024 (T321312)', diff saved to https://phabricator.wikimedia.org/P36660 and previous config saved to /var/cache/conftool/dbconfig/20221027-075433-ladsgroup.json [production]
07:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1024.eqiad.wmnet with reason: Maintenance [production]
07:54 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es1024.eqiad.wmnet with reason: Maintenance [production]
07:53 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd1005.eqiad.wmnet to plain [production]
07:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T318605)', diff saved to https://phabricator.wikimedia.org/P36659 and previous config saved to /var/cache/conftool/dbconfig/20221027-075327-ladsgroup.json [production]
07:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
07:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
07:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es2023', diff saved to https://phabricator.wikimedia.org/P36658 and previous config saved to /var/cache/conftool/dbconfig/20221027-075118-ladsgroup.json [production]
07:48 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd1005.eqiad.wmnet to drbd [production]
07:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P36657 and previous config saved to /var/cache/conftool/dbconfig/20221027-074521-ladsgroup.json [production]
07:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P36656 and previous config saved to /var/cache/conftool/dbconfig/20221027-074050-ladsgroup.json [production]
07:40 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1009.eqiad.wmnet [production]
07:39 <dcausse> restarting blazegraph on wdqs1016 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
07:38 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd1005.eqiad.wmnet to drbd [production]
07:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es2023 (T321312)', diff saved to https://phabricator.wikimedia.org/P36655 and previous config saved to /var/cache/conftool/dbconfig/20221027-073612-ladsgroup.json [production]
07:32 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1009.eqiad.wmnet [production]
07:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318950)', diff saved to https://phabricator.wikimedia.org/P36654 and previous config saved to /var/cache/conftool/dbconfig/20221027-073014-ladsgroup.json [production]
07:27 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1009.eqiad.wmnet with OS bullseye [production]
07:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318950)', diff saved to https://phabricator.wikimedia.org/P36653 and previous config saved to /var/cache/conftool/dbconfig/20221027-072543-ladsgroup.json [production]
07:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es2023 (T321312)', diff saved to https://phabricator.wikimedia.org/P36652 and previous config saved to /var/cache/conftool/dbconfig/20221027-072536-ladsgroup.json [production]
07:25 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2023.codfw.wmnet with reason: Maintenance [production]
07:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2023.codfw.wmnet with reason: Maintenance [production]
07:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2109 (T318950)', diff saved to https://phabricator.wikimedia.org/P36651 and previous config saved to /var/cache/conftool/dbconfig/20221027-072219-ladsgroup.json [production]
07:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
07:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
07:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318950)', diff saved to https://phabricator.wikimedia.org/P36650 and previous config saved to /var/cache/conftool/dbconfig/20221027-072157-ladsgroup.json [production]
07:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1157 (T318950)', diff saved to https://phabricator.wikimedia.org/P36649 and previous config saved to /var/cache/conftool/dbconfig/20221027-072148-ladsgroup.json [production]
07:21 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
07:21 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
07:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]