5251-5300 of 10000 results (65ms)
2022-06-13 ยง
21:51 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance [production]
21:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance [production]
21:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T310011)', diff saved to https://phabricator.wikimedia.org/P29691 and previous config saved to /var/cache/conftool/dbconfig/20220613-215118-marostegui.json [production]
21:48 <mutante> gitlab-runner* - sequentially pausing, rebooting, resuming one by one [production]
21:44 <mutante> gitlab-runner1001 - pause from accepting jobs - rebooting [production]
21:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P29690 and previous config saved to /var/cache/conftool/dbconfig/20220613-213613-marostegui.json [production]
21:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P29689 and previous config saved to /var/cache/conftool/dbconfig/20220613-212108-marostegui.json [production]
21:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T310011)', diff saved to https://phabricator.wikimedia.org/P29688 and previous config saved to /var/cache/conftool/dbconfig/20220613-210603-marostegui.json [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:29 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:29 <cjming> end of UTC late backport window [production]
20:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:27 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:805206|Disable TOC A/B test for beta cluster (T309683)]] (duration: 03m 29s) [production]
20:22 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800857|ugwiki: Add localized mobile wordmark (T309431)]] (duration: 03m 30s) [production]
20:19 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1050.eqiad.wmnet [production]
20:18 <cjming@deploy1002> Synchronized static/images/mobile/copyright/wikipedia-wordmark-ug.svg: Config: [[gerrit:800857|ugwiki: Add localized mobile wordmark (T309431)]] (duration: 03m 36s) [production]
20:18 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:17 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1121 (T310011)', diff saved to https://phabricator.wikimedia.org/P29687 and previous config saved to /var/cache/conftool/dbconfig/20220613-201420-marostegui.json [production]
20:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
20:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
20:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
20:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
20:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T310011)', diff saved to https://phabricator.wikimedia.org/P29686 and previous config saved to /var/cache/conftool/dbconfig/20220613-201407-marostegui.json [production]
20:12 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1050.eqiad.wmnet [production]
20:11 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800856|crhwiki: Add localized mobile wordmark (T309431)]] (duration: 03m 27s) [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:09 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:08 <cjming@deploy1002> Synchronized static/images/mobile/copyright/wikipedia-wordmark-crh.svg: Config: [[gerrit:800856|crhwiki: Add localized mobile wordmark (T309431)]] (duration: 03m 16s) [production]
20:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P29685 and previous config saved to /var/cache/conftool/dbconfig/20220613-195902-marostegui.json [production]
19:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P29684 and previous config saved to /var/cache/conftool/dbconfig/20220613-194356-marostegui.json [production]
19:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T310011)', diff saved to https://phabricator.wikimedia.org/P29683 and previous config saved to /var/cache/conftool/dbconfig/20220613-192851-marostegui.json [production]
19:12 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on etherpad1003.eqiad.wmnet with reason: kernel upgrade [production]
19:12 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on etherpad1003.eqiad.wmnet with reason: kernel upgrade [production]
19:11 <mutante> etherpad - minimal downtime - rebooting etherpad1003 [production]
19:07 <mutante> gerrit2002 - rebooting [production]
19:04 <mutante> gitlab2003 - rebooting [production]
19:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1141 (T310011)', diff saved to https://phabricator.wikimedia.org/P29682 and previous config saved to /var/cache/conftool/dbconfig/20220613-190314-marostegui.json [production]
19:03 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
19:03 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
19:01 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1049.eqiad.wmnet [production]
18:55 <mutante> gitlab2002 - rebooting [production]
18:40 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
18:40 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
18:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T310011)', diff saved to https://phabricator.wikimedia.org/P29681 and previous config saved to /var/cache/conftool/dbconfig/20220613-184015-marostegui.json [production]