4401-4450 of 10000 results (56ms)
2022-01-31 ยง
20:10 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:10 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:09 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:09 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:07 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:07 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P19706 and previous config saved to /var/cache/conftool/dbconfig/20220131-195614-marostegui.json [production]
19:42 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P19705 and previous config saved to /var/cache/conftool/dbconfig/20220131-194109-marostegui.json [production]
19:29 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:28 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:27 <cjming> end of UTC evening backport & config window [production]
19:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298559)', diff saved to https://phabricator.wikimedia.org/P19704 and previous config saved to /var/cache/conftool/dbconfig/20220131-192604-marostegui.json [production]
19:24 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:757735|Disable A/B test (T297924)]] (duration: 00m 49s) [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:20 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:17 <cjming@deploy1002> Synchronized wmf-config/config: Config: [[gerrit:757500|Update config for idwiki: (T299676)]] (duration: 00m 50s) [production]
19:14 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T298559)', diff saved to https://phabricator.wikimedia.org/P19703 and previous config saved to /var/cache/conftool/dbconfig/20220131-191356-marostegui.json [production]
19:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
19:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
19:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298559)', diff saved to https://phabricator.wikimedia.org/P19702 and previous config saved to /var/cache/conftool/dbconfig/20220131-191348-marostegui.json [production]
19:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:10 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 411af378c606c0f987679a1eebd901326dd5db18: [wmf-config]: Undeploy gdi survey from cawiki in production (T300544) (duration: 00m 50s) [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:06 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: a659cb0089da0c6d501263c19dd692a286601d2c: Revert "commonswiki: Add leg.journals.isu.ac.ir to the wgCopyUploadsDomains allowlist" (T300217) (duration: 00m 50s) [production]
18:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P19701 and previous config saved to /var/cache/conftool/dbconfig/20220131-185843-marostegui.json [production]
18:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P19700 and previous config saved to /var/cache/conftool/dbconfig/20220131-184339-marostegui.json [production]
18:41 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup1004.eqiad.wmnet with OS buster [production]
18:41 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS buster [production]
18:40 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1003.eqiad.wmnet with OS buster [production]
18:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298558)', diff saved to https://phabricator.wikimedia.org/P19699 and previous config saved to /var/cache/conftool/dbconfig/20220131-184006-marostegui.json [production]
18:28 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298559)', diff saved to https://phabricator.wikimedia.org/P19698 and previous config saved to /var/cache/conftool/dbconfig/20220131-182834-marostegui.json [production]
18:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1149 (T298559)', diff saved to https://phabricator.wikimedia.org/P19697 and previous config saved to /var/cache/conftool/dbconfig/20220131-182728-marostegui.json [production]
18:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
18:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
18:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T298559)', diff saved to https://phabricator.wikimedia.org/P19696 and previous config saved to /var/cache/conftool/dbconfig/20220131-182719-marostegui.json [production]
18:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P19695 and previous config saved to /var/cache/conftool/dbconfig/20220131-182501-marostegui.json [production]
18:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P19694 and previous config saved to /var/cache/conftool/dbconfig/20220131-181215-marostegui.json [production]
18:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P19693 and previous config saved to /var/cache/conftool/dbconfig/20220131-180956-marostegui.json [production]
18:07 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:04 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
18:02 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster [production]
18:01 <moritzm> installing NSS security updates [production]