|
2022-11-07
ยง
|
| 07:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38182 and previous config saved to /var/cache/conftool/dbconfig/20221107-074938-marostegui.json |
[production] |
| 07:47 |
<marostegui@deploy1002> |
Finished scap: Backport for [[gerrit:853528|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] (duration: 04m 53s) |
[production] |
| 07:46 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:44 |
<urbanecm> |
Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=frwiki` in a tmux at mwmaint1002 (T318457) |
[production] |
| 07:42 |
<marostegui@deploy1002> |
marostegui and marostegui: Backport for [[gerrit:853528|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 07:42 |
<marostegui@deploy1002> |
Started scap: Backport for [[gerrit:853528|Revert "ProductionServices.php: Promote pc1014 to pc1 master"]] |
[production] |
| 07:37 |
<elukey> |
`elukey@aux-k8s-worker1002:~$ sudo systemctl reset-failed ifup@ens13.service` |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38181 and previous config saved to /var/cache/conftool/dbconfig/20221107-073431-marostegui.json |
[production] |
| 07:31 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P38180 and previous config saved to /var/cache/conftool/dbconfig/20221107-071925-marostegui.json |
[production] |
| 07:17 |
<marostegui@deploy1002> |
Finished scap: Backport for [[gerrit:853711|ProductionServices.php: Promote pc1014 to pc1 master (T322295)]] (duration: 04m 29s) |
[production] |
| 07:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:17 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:13 |
<marostegui@deploy1002> |
marostegui and marostegui: Backport for [[gerrit:853711|ProductionServices.php: Promote pc1014 to pc1 master (T322295)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 07:13 |
<marostegui@deploy1002> |
Started scap: Backport for [[gerrit:853711|ProductionServices.php: Promote pc1014 to pc1 master (T322295)]] |
[production] |
| 07:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:07 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover |
[production] |
| 07:07 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on pc2011.codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Primary switchover |
[production] |
| 07:05 |
<urbanecm> |
Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=bnwiki` in a tmux at mwmaint1002 (T318457) |
[production] |
| 07:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38179 and previous config saved to /var/cache/conftool/dbconfig/20221107-070418-marostegui.json |
[production] |
| 07:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1141 (T321123)', diff saved to https://phabricator.wikimedia.org/P38178 and previous config saved to /var/cache/conftool/dbconfig/20221107-070311-marostegui.json |
[production] |
| 07:03 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38177 and previous config saved to /var/cache/conftool/dbconfig/20221107-070249-marostegui.json |
[production] |
| 07:02 |
<urbanecm> |
Run `time mwscript extensions/GrowthExperiments/maintenance/updateIsActiveFlagForMentees.php --wiki=cswiki` in a tmux at mwmaint1002 (T318457) |
[production] |
| 07:01 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:853509|Add support for gemm_mentee_is_active (T318457)]], [[gerrit:853440|Calculate mentorship-related metrics (T318684)]] (duration: 06m 27s) |
[production] |
| 06:55 |
<urbanecm@deploy1002> |
urbanecm and urbanecm: Backport for [[gerrit:853509|Add support for gemm_mentee_is_active (T318457)]], [[gerrit:853440|Calculate mentorship-related metrics (T318684)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 06:54 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:853509|Add support for gemm_mentee_is_active (T318457)]], [[gerrit:853440|Calculate mentorship-related metrics (T318684)]] |
[production] |
| 06:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2024 T322406', diff saved to https://phabricator.wikimedia.org/P38176 and previous config saved to /var/cache/conftool/dbconfig/20221107-065251-root.json |
[production] |
| 06:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es2023 to es5 primary and set section read-write T322406', diff saved to https://phabricator.wikimedia.org/P38175 and previous config saved to /var/cache/conftool/dbconfig/20221107-065048-root.json |
[production] |
| 06:49 |
<marostegui> |
Starting es5 codfw failover from es2024 to es2023 - T322406 |
[production] |
| 06:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38174 and previous config saved to /var/cache/conftool/dbconfig/20221107-064743-marostegui.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406 |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set es2023 with weight 0 T322406', diff saved to https://phabricator.wikimedia.org/P38173 and previous config saved to /var/cache/conftool/dbconfig/20221107-064608-root.json |
[production] |
| 06:45 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322406 |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P38172 and previous config saved to /var/cache/conftool/dbconfig/20221107-063236-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38171 and previous config saved to /var/cache/conftool/dbconfig/20221107-061730-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1121 (T321123)', diff saved to https://phabricator.wikimedia.org/P38170 and previous config saved to /var/cache/conftool/dbconfig/20221107-061019-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:10 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 3292 |
[production] |
| 06:09 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:09 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:09 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1121.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:09 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 3292 |
[production] |