|
2022-08-04
ยง
|
| 19:44 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for 8 hosts |
[production] |
| 19:42 |
<Emperor> |
rebooting thanos-be2001 to fix drive ordering |
[production] |
| 19:37 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for elastic2071.codfw.wmnet |
[production] |
| 19:37 |
<bking@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for elastic2071.codfw.wmnet |
[production] |
| 19:31 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic2071.codfw.wmnet with reason: T310146 |
[production] |
| 19:31 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic2071.codfw.wmnet with reason: T310146 |
[production] |
| 19:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:12 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 19:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:12 |
<ryankemper@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 19:11 |
<ryankemper@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 19:11 |
<dancy> |
There were many errors during php-fpm restart due to failure to contact http://lvs2009:9090/pools/appservers-https_443/mw2361.codfw.wmnet and the like. |
[production] |
| 19:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 19:10 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.39.0-wmf.23 refs T308076 |
[production] |
| 19:09 |
<ryankemper@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop: apply |
[production] |
| 19:09 |
<ryankemper@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop: apply |
[production] |
| 19:05 |
<otto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 19:04 |
<otto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 19:04 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 19:03 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 19:03 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 19:02 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 19:02 |
<ottomata> |
roll-restarting eventgate-analytics-external to pick up backwards incompatible schema change - T314151 |
[production] |
| 18:47 |
<ryankemper@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 18:46 |
<ryankemper@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 18:41 |
<cwhite> |
poweroff kafka-logging2003 - T310145 |
[production] |
| 18:39 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw237[0-6].codfw.wmnet |
[production] |
| 18:39 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts |
[production] |
| 18:39 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for 7 hosts |
[production] |
| 18:35 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw2369.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for mw2369.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw2368.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for mw2368.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw2367.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for mw2367.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw2369.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw2368.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw2367.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw2366.codfw.wmnet |
[production] |
| 18:35 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for mw2366.codfw.wmnet |
[production] |
| 18:34 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw2366.codfw.wmnet |
[production] |
| 18:30 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw2279.codfw.wmnet |
[production] |
| 18:30 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw2278.codfw.wmnet |
[production] |
| 18:29 |
<dzahn@cumin2002> |
conftool action : set/pooled=yes; selector: dc=codfw,name=mw2277.codfw.wmnet |
[production] |
| 18:29 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw2276.codfw.wmnet |
[production] |
| 18:29 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for mw2276.codfw.wmnet |
[production] |
| 18:29 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw2275.codfw.wmnet |
[production] |
| 18:29 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for mw2275.codfw.wmnet |
[production] |
| 18:29 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mw2274.codfw.wmnet |
[production] |
| 18:29 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for mw2274.codfw.wmnet |
[production] |