2251-2300 of 10000 results (53ms)
2022-04-26 ยง
23:35 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
23:35 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
23:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P26642 and previous config saved to /var/cache/conftool/dbconfig/20220426-233545-ladsgroup.json [production]
23:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26641 and previous config saved to /var/cache/conftool/dbconfig/20220426-232447-ladsgroup.json [production]
23:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26640 and previous config saved to /var/cache/conftool/dbconfig/20220426-232040-ladsgroup.json [production]
23:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26639 and previous config saved to /var/cache/conftool/dbconfig/20220426-230942-ladsgroup.json [production]
23:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26638 and previous config saved to /var/cache/conftool/dbconfig/20220426-230535-ladsgroup.json [production]
22:57 <tgr@deploy1002> Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/extension.json: Backport: [[gerrit:786366|Enable SkinAddFooterLinks hook]] (duration: 00m 51s) [production]
22:56 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:56 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:56 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
22:56 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
22:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26637 and previous config saved to /var/cache/conftool/dbconfig/20220426-225437-ladsgroup.json [production]
22:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1113:3316 (T298556)', diff saved to https://phabricator.wikimedia.org/P26636 and previous config saved to /var/cache/conftool/dbconfig/20220426-225326-ladsgroup.json [production]
22:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
22:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
22:51 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:51 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:51 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
22:51 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
22:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P26635 and previous config saved to /var/cache/conftool/dbconfig/20220426-225030-ladsgroup.json [production]
22:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1119 (T306560)', diff saved to https://phabricator.wikimedia.org/P26634 and previous config saved to /var/cache/conftool/dbconfig/20220426-224757-ladsgroup.json [production]
22:47 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
22:47 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
22:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T306560)', diff saved to https://phabricator.wikimedia.org/P26633 and previous config saved to /var/cache/conftool/dbconfig/20220426-224749-ladsgroup.json [production]
22:46 <tgr@deploy1002> Finished scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]] (duration: 21m 40s) [production]
22:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26632 and previous config saved to /var/cache/conftool/dbconfig/20220426-223244-ladsgroup.json [production]
22:25 <tgr@deploy1002> Started scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]] [production]
22:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26631 and previous config saved to /var/cache/conftool/dbconfig/20220426-221739-ladsgroup.json [production]
22:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T306560)', diff saved to https://phabricator.wikimedia.org/P26630 and previous config saved to /var/cache/conftool/dbconfig/20220426-220234-ladsgroup.json [production]
22:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1135 (T306560)', diff saved to https://phabricator.wikimedia.org/P26629 and previous config saved to /var/cache/conftool/dbconfig/20220426-220001-ladsgroup.json [production]
22:00 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
21:59 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
21:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134 (T306560)', diff saved to https://phabricator.wikimedia.org/P26628 and previous config saved to /var/cache/conftool/dbconfig/20220426-215953-ladsgroup.json [production]
21:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26627 and previous config saved to /var/cache/conftool/dbconfig/20220426-214448-ladsgroup.json [production]
21:38 <aqu@deploy1002> Finished deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9] (duration: 00m 07s) [production]
21:37 <aqu@deploy1002> Started deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9] [production]
21:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26626 and previous config saved to /var/cache/conftool/dbconfig/20220426-212943-ladsgroup.json [production]
21:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134 (T306560)', diff saved to https://phabricator.wikimedia.org/P26625 and previous config saved to /var/cache/conftool/dbconfig/20220426-211437-ladsgroup.json [production]
21:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1134 (T306560)', diff saved to https://phabricator.wikimedia.org/P26624 and previous config saved to /var/cache/conftool/dbconfig/20220426-211204-ladsgroup.json [production]
21:12 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
21:12 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance [production]
21:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T306560)', diff saved to https://phabricator.wikimedia.org/P26623 and previous config saved to /var/cache/conftool/dbconfig/20220426-211156-ladsgroup.json [production]
21:05 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:05 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:03 <dzahn@cumin2002> conftool action : set/weight=30; selector: dc=codfw,name=mw2418.codfw.wmnet [production]
21:03 <dzahn@cumin2002> conftool action : set/weight=30; selector: dc=codfw,name=mw2417.codfw.wmnet [production]
21:02 <dzahn@cumin2002> conftool action : set/weight=30; selector: dc=codfw,name=mw2416.codfw.wmnet [production]