3401-3450 of 10000 results (96ms)
2022-07-25 §
09:14 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
09:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 15 hosts with reason: Maintenance [production]
09:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 15 hosts with reason: Maintenance [production]
09:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2079.codfw.wmnet with reason: Maintenance [production]
09:12 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2079.codfw.wmnet with reason: Maintenance [production]
09:12 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1111.eqiad.wmnet with reason: Maintenance [production]
09:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1111.eqiad.wmnet with reason: Maintenance [production]
09:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
09:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
09:10 <ladsgroup@cumin1001> END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
09:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
09:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1180 (T312863)', diff saved to https://phabricator.wikimedia.org/P31821 and previous config saved to /var/cache/conftool/dbconfig/20220725-090906-ladsgroup.json [production]
09:09 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
09:08 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
09:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1123 (T312863)', diff saved to https://phabricator.wikimedia.org/P31820 and previous config saved to /var/cache/conftool/dbconfig/20220725-090604-ladsgroup.json [production]
09:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
09:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
09:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 100%: Maint done', diff saved to https://phabricator.wikimedia.org/P31819 and previous config saved to /var/cache/conftool/dbconfig/20220725-090113-ladsgroup.json [production]
08:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 75%: Maint done', diff saved to https://phabricator.wikimedia.org/P31818 and previous config saved to /var/cache/conftool/dbconfig/20220725-084609-ladsgroup.json [production]
08:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 10%: Maint done', diff saved to https://phabricator.wikimedia.org/P31817 and previous config saved to /var/cache/conftool/dbconfig/20220725-083105-ladsgroup.json [production]
08:17 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
08:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 5%: Maint done', diff saved to https://phabricator.wikimedia.org/P31816 and previous config saved to /var/cache/conftool/dbconfig/20220725-081601-ladsgroup.json [production]
08:15 <kartik@deploy1002> Synchronized php-1.39.0-wmf.21/extensions/Translate: Backport: [[gerrit:816272|ReviewTranslationActionApi: Move to namespace and add strict types (T312008 T313608)]] (duration: 03m 09s) [production]
08:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
08:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
08:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:40 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:37 <kartik@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:802443|Explicitly set math rendering modes (T309686)]] (duration: 03m 11s) [production]
07:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:31 <volans@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:28 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:27 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:23 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
07:16 <kartik@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:815829|Enable Section Translation in Uzbek Wikipedia (T310116)]] (duration: 03m 04s) [production]
07:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:15 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
06:30 <XioNoX> power off asw2-d5-eqiad for decommissioning - T313115 [production]
2022-07-24 §
20:54 <btullis@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org [production]
20:37 <btullis@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org [production]
14:43 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
14:42 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
10:02 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
10:02 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
10:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129 (T312863)', diff saved to https://phabricator.wikimedia.org/P31815 and previous config saved to /var/cache/conftool/dbconfig/20220724-100221-ladsgroup.json [production]