|
2022-07-28
§
|
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2142 T313811', diff saved to https://phabricator.wikimedia.org/P32026 and previous config saved to /var/cache/conftool/dbconfig/20220728-060757-root.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db2144 to x2 primary T313811', diff saved to https://phabricator.wikimedia.org/P32025 and previous config saved to /var/cache/conftool/dbconfig/20220728-060057-marostegui.json |
[production] |
| 06:00 |
<marostegui> |
Starting x2 codfw failover from db2142 to db2144 - T313811 |
[production] |
| 05:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover x2 T313811 |
[production] |
| 05:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover x2 T313811 |
[production] |
| 03:28 |
<ejegg> |
updated fundraising CiviCRM from e0962be6 to 3143dda9 |
[production] |
| 01:28 |
<tstarling@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: move OAuth token storage T313578 (duration: 03m 04s) |
[production] |
| 01:27 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:18 |
<tstarling@deploy1002> |
Synchronized php-1.39.0-wmf.21/extensions/OAuth: New config var for T313578, not yet used (duration: 03m 23s) |
[production] |
| 01:11 |
<tstarling@deploy1002> |
Synchronized php-1.39.0-wmf.22/extensions/OAuth: New config var for T313578, not yet used (duration: 03m 39s) |
[production] |
|
2022-07-27
§
|
| 23:59 |
<tstarling@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: sync again now that scap proxy list is fixed T313730 T313496 (duration: 03m 25s) |
[production] |
| 23:49 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:49 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:49 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:48 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:45 |
<tstarling@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: move CentralAuth sessions to Kask T313496 (duration: 05m 34s) |
[production] |
| 23:45 |
<rzl@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[2251-2255,2257-2258].codfw.wmnet |
[production] |
| 23:45 |
<rzl@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 23:43 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:38 |
<rzl@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 23:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:35 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:35 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:29 |
<tstarling@deploy1002> |
Synchronized wmf-config/CommonSettings.php: increase wgObjectCacheSessionExpiry to 86400 (duration: 03m 30s) |
[production] |
| 23:29 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 23:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 23:26 |
<rzl@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts mw[2251-2255,2257-2258].codfw.wmnet |
[production] |
| 23:18 |
<rzl@cumin2002> |
conftool action : set/pooled=inactive; selector: name=mw225[1-57-8].codfw.wmnet |
[production] |
| 23:17 |
<rzl@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 7 hosts with reason: Decom |
[production] |
| 23:17 |
<rzl@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 7 hosts with reason: Decom |
[production] |
| 23:14 |
<tstarling@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply |
[production] |
| 23:14 |
<tstarling@deploy1002> |
helmfile [eqiad] START helmfile.d/services/sessionstore: apply |
[production] |
| 23:14 |
<tstarling@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/sessionstore: apply |
[production] |
| 23:13 |
<tstarling@deploy1002> |
helmfile [codfw] START helmfile.d/services/sessionstore: apply |
[production] |
| 23:13 |
<rzl@cumin2002> |
conftool action : set/pooled=no; selector: name=mw225[1-57-8].codfw.wmnet |
[production] |
| 23:08 |
<tstarling@deploy1002> |
helmfile [staging] DONE helmfile.d/services/sessionstore: apply |
[production] |
| 23:08 |
<tstarling@deploy1002> |
helmfile [staging] START helmfile.d/services/sessionstore: apply |
[production] |
| 22:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 22:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 22:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 22:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |