|
2022-03-29
§
|
| 05:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1134.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T300775)', diff saved to https://phabricator.wikimedia.org/P23440 and previous config saved to /var/cache/conftool/dbconfig/20220329-051943-marostegui.json |
[production] |
| 05:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P23439 and previous config saved to /var/cache/conftool/dbconfig/20220329-050438-marostegui.json |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1123 with weight 0 T301850', diff saved to https://phabricator.wikimedia.org/P23438 and previous config saved to /var/cache/conftool/dbconfig/20220329-050234-root.json |
[production] |
| 05:02 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Primary switchover s3 T301850 |
[production] |
| 05:02 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 20 hosts with reason: Primary switchover s3 T301850 |
[production] |
| 04:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P23437 and previous config saved to /var/cache/conftool/dbconfig/20220329-044933-marostegui.json |
[production] |
| 04:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T300775)', diff saved to https://phabricator.wikimedia.org/P23436 and previous config saved to /var/cache/conftool/dbconfig/20220329-043428-marostegui.json |
[production] |
| 02:30 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:30 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 02:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:29 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 02:08 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:08 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 02:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 02:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
|
2022-03-28
§
|
| 23:15 |
<eileen> |
civicrm revision 15d22bd1 -> 1c5d10e1 |
[production] |
| 23:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1184 (T300775)', diff saved to https://phabricator.wikimedia.org/P23434 and previous config saved to /var/cache/conftool/dbconfig/20220328-230012-marostegui.json |
[production] |
| 23:00 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1164 (T300775)', diff saved to https://phabricator.wikimedia.org/P23433 and previous config saved to /var/cache/conftool/dbconfig/20220328-230004-marostegui.json |
[production] |
| 22:52 |
<ejegg> |
updated fundraising python tools from 409c80b7 to 8f5119f6 |
[production] |
| 22:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P23431 and previous config saved to /var/cache/conftool/dbconfig/20220328-224459-marostegui.json |
[production] |
| 22:39 |
<rzl> |
rzl@cumin2002:~$ sudo cumin A:mw 'enable-puppet T205361' |
[production] |
| 22:31 |
<rzl> |
rzl@cumin2002:~$ sudo cumin A:mw 'disable-puppet T205361' |
[production] |
| 22:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P23430 and previous config saved to /var/cache/conftool/dbconfig/20220328-222953-marostegui.json |
[production] |
| 22:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1164 (T300775)', diff saved to https://phabricator.wikimedia.org/P23429 and previous config saved to /var/cache/conftool/dbconfig/20220328-221448-marostegui.json |
[production] |
| 21:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:32 |
<sbassett> |
Undeployed sec patch for T285159, which caused a high volume of errors on the canaries |
[production] |
| 21:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:27 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:12 |
<eileen> |
civicrm revision 4e5b37c3 -> 15d22bd1 |
[production] |
| 21:09 |
<eileen> |
tools revision changed from d1d7b100 to 409c80b7 |
[production] |
| 21:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:06 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:06 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:06 |
<eileen> |
revision changed from d1d7b100 to 409c80b7 |
[production] |
| 21:05 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:03 |
<sbassett@deploy1002> |
Synchronized wmf-config/CommonSettings-labs.php: Deploy CS-labs.php config to set StopForumSpam to enforce on beta (duration: 01m 03s) |
[production] |
| 20:40 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:39 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |