|
2022-03-29
ยง
|
| 20:52 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:51 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:50 |
<catrope@deploy1002> |
Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. |
[production] |
| 20:46 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23622 and previous config saved to /var/cache/conftool/dbconfig/20220329-204241-ladsgroup.json |
[production] |
| 20:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23621 and previous config saved to /var/cache/conftool/dbconfig/20220329-204034-ladsgroup.json |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23620 and previous config saved to /var/cache/conftool/dbconfig/20220329-204021-ladsgroup.json |
[production] |
| 20:39 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773302|Add wikimedia.com to wgNoFollowDomainExceptions (T304555)]] (duration: 01m 06s) |
[production] |
| 20:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P23619 and previous config saved to /var/cache/conftool/dbconfig/20220329-203838-ladsgroup.json |
[production] |
| 20:38 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:36 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:774953|[config]: Deploy gdi-safety-survey to ES,EN,FR and PT wikis]] (duration: 00m 56s) |
[production] |
| 20:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:30 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773896|Config for new android schemas]] (duration: 01m 00s) |
[production] |
| 20:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23618 and previous config saved to /var/cache/conftool/dbconfig/20220329-202516-ladsgroup.json |
[production] |
| 20:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P23617 and previous config saved to /var/cache/conftool/dbconfig/20220329-202333-ladsgroup.json |
[production] |
| 20:20 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 20:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131 (T297189)', diff saved to https://phabricator.wikimedia.org/P23616 and previous config saved to /var/cache/conftool/dbconfig/20220329-201611-marostegui.json |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P23615 and previous config saved to /var/cache/conftool/dbconfig/20220329-201041-ladsgroup.json |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23614 and previous config saved to /var/cache/conftool/dbconfig/20220329-201011-ladsgroup.json |
[production] |
| 20:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P23613 and previous config saved to /var/cache/conftool/dbconfig/20220329-200106-marostegui.json |
[production] |
| 19:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23612 and previous config saved to /var/cache/conftool/dbconfig/20220329-195505-ladsgroup.json |
[production] |
| 19:48 |
<eileen> |
civicrm revision changed from 1c5d10e1 to 951ffb1d |
[production] |
| 19:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P23611 and previous config saved to /var/cache/conftool/dbconfig/20220329-194601-marostegui.json |
[production] |
| 19:43 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@8e9f97c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@8e9f97c] (duration: 07m 17s) |
[production] |
| 19:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23610 and previous config saved to /var/cache/conftool/dbconfig/20220329-194256-ladsgroup.json |
[production] |
| 19:42 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:42 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23609 and previous config saved to /var/cache/conftool/dbconfig/20220329-194248-ladsgroup.json |
[production] |
| 19:40 |
<moritzm> |
uploaded cachelib 0.4.1-2~wmf1 to bullseye-wikimedia T301638 |
[production] |
| 19:35 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@8e9f97c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@8e9f97c] |
[production] |
| 19:35 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@8e9f97c] (thin): Regular analytics weekly train THIN [analytics/refinery@8e9f97c] (duration: 00m 08s) |
[production] |
| 19:35 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@8e9f97c] (thin): Regular analytics weekly train THIN [analytics/refinery@8e9f97c] |
[production] |
| 19:35 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@8e9f97c]: Regular analytics weekly train [analytics/refinery@8e9f97c] (duration: 21m 13s) |
[production] |
| 19:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131 (T297189)', diff saved to https://phabricator.wikimedia.org/P23608 and previous config saved to /var/cache/conftool/dbconfig/20220329-193055-marostegui.json |
[production] |