4201-4250 of 10000 results (62ms)
2022-05-05 ยง
07:42 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:41 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:39 <tgr> running extensions/GrowthExperiments/maintenance/changeWikiConfig.php for T306792 [production]
07:38 <tgr@deploy1002> Synchronized php-1.39.0-wmf.9/extensions/GrowthExperiments: Backport: [[gerrit:789185|Community configuration: Allow writing sub-fields programmatically (T306792)]] (duration: 00m 52s) [production]
07:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P27644 and previous config saved to /var/cache/conftool/dbconfig/20220505-073826-ladsgroup.json [production]
07:36 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P27643 and previous config saved to /var/cache/conftool/dbconfig/20220505-073543-ladsgroup.json [production]
07:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:34 <tgr@deploy1002> Synchronized php-1.39.0-wmf.10/extensions/GrowthExperiments: Backport: [[gerrit:789326|Community configuration: Allow writing sub-fields programmatically (T306792)]] (duration: 00m 54s) [production]
07:23 <moritzm> powercycling restbase1018 [production]
07:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T307525)', diff saved to https://phabricator.wikimedia.org/P27642 and previous config saved to /var/cache/conftool/dbconfig/20220505-072321-ladsgroup.json [production]
07:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T307525)', diff saved to https://phabricator.wikimedia.org/P27641 and previous config saved to /var/cache/conftool/dbconfig/20220505-072038-ladsgroup.json [production]
07:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
07:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
07:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1168 (T307525)', diff saved to https://phabricator.wikimedia.org/P27640 and previous config saved to /var/cache/conftool/dbconfig/20220505-071911-ladsgroup.json [production]
07:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
07:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
07:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T307525)', diff saved to https://phabricator.wikimedia.org/P27639 and previous config saved to /var/cache/conftool/dbconfig/20220505-071904-ladsgroup.json [production]
07:13 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host restbase1018.eqiad.wmnet [production]
07:07 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[1018-1019].eqiad.wmnet with reason: reboot [production]
07:07 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[1018-1019].eqiad.wmnet with reason: reboot [production]
07:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1157 (T307525)', diff saved to https://phabricator.wikimedia.org/P27638 and previous config saved to /var/cache/conftool/dbconfig/20220505-070422-ladsgroup.json [production]
07:04 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
07:04 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
07:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P27637 and previous config saved to /var/cache/conftool/dbconfig/20220505-070357-ladsgroup.json [production]
06:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P27636 and previous config saved to /var/cache/conftool/dbconfig/20220505-064852-ladsgroup.json [production]
06:40 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance [production]
06:40 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance [production]
06:40 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
06:40 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
06:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136 (T307525)', diff saved to https://phabricator.wikimedia.org/P27635 and previous config saved to /var/cache/conftool/dbconfig/20220505-064029-ladsgroup.json [production]
06:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T307525)', diff saved to https://phabricator.wikimedia.org/P27634 and previous config saved to /var/cache/conftool/dbconfig/20220505-063347-ladsgroup.json [production]
06:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1180 (T307525)', diff saved to https://phabricator.wikimedia.org/P27633 and previous config saved to /var/cache/conftool/dbconfig/20220505-062935-ladsgroup.json [production]
06:29 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
06:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
06:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T307525)', diff saved to https://phabricator.wikimedia.org/P27632 and previous config saved to /var/cache/conftool/dbconfig/20220505-062927-ladsgroup.json [production]
06:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P27631 and previous config saved to /var/cache/conftool/dbconfig/20220505-062524-ladsgroup.json [production]
06:19 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
06:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
06:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27630 and previous config saved to /var/cache/conftool/dbconfig/20220505-061421-ladsgroup.json [production]
06:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P27629 and previous config saved to /var/cache/conftool/dbconfig/20220505-061019-ladsgroup.json [production]
05:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27628 and previous config saved to /var/cache/conftool/dbconfig/20220505-055916-ladsgroup.json [production]
05:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136 (T307525)', diff saved to https://phabricator.wikimedia.org/P27627 and previous config saved to /var/cache/conftool/dbconfig/20220505-055514-ladsgroup.json [production]
05:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T307525)', diff saved to https://phabricator.wikimedia.org/P27625 and previous config saved to /var/cache/conftool/dbconfig/20220505-054411-ladsgroup.json [production]
05:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1127', diff saved to https://phabricator.wikimedia.org/P27624 and previous config saved to /var/cache/conftool/dbconfig/20220505-054027-marostegui.json [production]
05:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1136 (T307525)', diff saved to https://phabricator.wikimedia.org/P27623 and previous config saved to /var/cache/conftool/dbconfig/20220505-054013-ladsgroup.json [production]
05:40 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]