|
2021-09-14
§
|
| 09:09 |
<Emperor> |
swift rebalance to remove h/w faulty host ms-be2045 T290881 |
[production] |
| 09:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:57 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:47 |
<moritzm> |
installing testvm2002 |
[production] |
| 08:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2002.codfw.wmnet |
[production] |
| 08:28 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host testvm2002.codfw.wmnet |
[production] |
| 08:27 |
<hashar@deploy1002> |
Started scap: testwikis wikis to 1.37.0-wmf.23 |
[production] |
| 08:25 |
<godog> |
poweroff ms-be2045 and set it as failed in netbox - T290881 |
[production] |
| 08:24 |
<hashar> |
train: applied security patches for 1.37.0-wmf.23 # T281164 |
[production] |
| 08:05 |
<godog> |
wipe non-os partitions from ms-be2045 - T290881 |
[production] |
| 07:50 |
<vgutierrez> |
update acme-chief to version 0.31 on acmechief hosts - T290249 |
[production] |
| 04:47 |
<eileen> |
civicrm revision changed from 1f071f6c6c to e6bf81d99c, config revision is 23eda8ba3a |
[production] |
| 02:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:39 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:07 |
<James_F> |
wmf/1.37.0-wmf.23 was branched at ea72c9b690c2159a12beec2f518b61cc499ed521 for T281164 |
[production] |
| 02:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
|
2021-09-13
§
|
| 23:54 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:52 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:45 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: T290759: Undeploy VipsScaler: III – Don't set wmgUseVips, now ignored (duration: 00m 58s) |
[production] |
| 23:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:43 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:41 |
<jforrester@deploy1002> |
Synchronized wmf-config/CommonSettings.php: T290759: Undeploy VipsScaler: II – Don't load regardless of config (duration: 00m 58s) |
[production] |
| 19:52 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: T290759 Undeploy VipsScaler: I – Disable on all wikis (duration: 00m 57s) |
[production] |
| 19:49 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:47 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:59 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:59 |
<urbanecm> |
[urbanecm@mwmaint2002 ~]$ mwscript resetAuthenticationThrottle.php --wiki={cswiki,cswikiversity} --signup --ip=185.47.223.49 # T290809 |
[production] |
| 18:58 |
<urbanecm@deploy1002> |
Synchronized wmf-config/throttle.php: 9db1d1ac938ca053c82fed88c8b6e75f97a52416: Add throttle rule for Czech wiki course (T290809) (duration: 00m 58s) |
[production] |
| 18:29 |
<ryankemper> |
[Cirrus] `eqiad` fully recovered (100% of shards), `codfw` at 99.816%. `codfw` is getting held up by recovery of `enwiki` shards which tend to be quite large |
[production] |
| 18:25 |
<razzi> |
reenable replication on dbstore1007 for T290841 |
[production] |
| 18:16 |
<cwhite> |
apply high log volume from ES mitigations to deprecated inputs |
[production] |
| 18:13 |
<razzi> |
razzi@dbstore1007:~$ sudo systemctl restart mariadb@s3.service for T290841 |
[production] |
| 18:05 |
<razzi> |
sudo systemctl restart mariadb@s2.service |
[production] |
| 17:48 |
<ryankemper> |
[Cirrus] `eqiad` is at 99.13% shards recovered and `codfw` is at 98.83% |
[production] |
| 17:20 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host sretest1002.eqiad.wmnet |
[production] |
| 17:17 |
<ryankemper> |
[Cirrus] `enwiki` searches appear to be working now. `production-search-eqiad` is at 93.5% recovered shards, `production-search-codfw` is at 95.3% recovered |
[production] |
| 16:57 |
<volans@cumin1001> |
START - Cookbook sre.experimental.reimage for host sretest1002.eqiad.wmnet |
[production] |
| 16:18 |
<legoktm@cumin1001> |
conftool action : set/pooled=false; selector: name=codfw,dnsdisc=eventgate-main |
[production] |
| 16:16 |
<volans@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1414.* |
[production] |
| 16:08 |
<volans@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1414.* |
[production] |
| 16:06 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host mw1414.eqiad.wmnet |
[production] |
| 15:54 |
<moritzm> |
filtered mx2001 on the routers for reimage T286911 |
[production] |
| 15:43 |
<vgutierrez> |
update acme-chief to version 0.31 on acmechief-test hosts - T290249 |
[production] |
| 15:40 |
<vgutierrez> |
upload acme-chief 0.31 to apt.wm.o (buster) - T290249 |
[production] |
| 15:32 |
<jelto> |
Traffic: depool codfw from user traffic |
[production] |
| 15:26 |
<jelto@cumin2002> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |