|
2021-08-17
ยง
|
| 14:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:40 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:37 |
<kormat@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Promote pc2013 to primary of pc3 T284825 (duration: 00m 58s) |
[production] |
| 14:25 |
<jynus> |
running a full testwiki media backup on a single thread, single worker T262668 |
[production] |
| 14:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:21 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:20 |
<kormat@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Promote pc2012 to primary of pc2 T284825 (duration: 00m 59s) |
[production] |
| 13:53 |
<jynus> |
rolling restart of minio on backup server |
[production] |
| 13:51 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 13:06 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 11:29 |
<phuedx@deploy1002> |
Synchronized php-1.37.0-wmf.18/extensions/SecurePoll/includes/Jobs/TallyElectionJob.php: Backport: [[gerrit:713361|tallyElectionJob: Catch and log exceptions (T288361)]] (duration: 00m 58s) |
[production] |
| 11:16 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17038 and previous config saved to /var/cache/conftool/dbconfig/20210817-111629-mvernon.json |
[production] |
| 11:15 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 11:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:01 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17037 and previous config saved to /var/cache/conftool/dbconfig/20210817-110125-mvernon.json |
[production] |
| 10:46 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17035 and previous config saved to /var/cache/conftool/dbconfig/20210817-104622-mvernon.json |
[production] |
| 10:31 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17034 and previous config saved to /var/cache/conftool/dbconfig/20210817-103118-mvernon.json |
[production] |
| 10:07 |
<effie> |
enable puppet on mediawiki hosts |
[production] |
| 09:52 |
<mvernon@cumin2002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2121.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:50 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2121.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:20 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 depooling: reimage to buster T288244', diff saved to https://phabricator.wikimedia.org/P17033 and previous config saved to /var/cache/conftool/dbconfig/20210817-092045-mvernon.json |
[production] |
| 09:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1456.eqiad.wmnet |
[production] |
| 09:16 |
<Emperor> |
reimaging db2121 to buster T288244 |
[production] |
| 09:08 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw1456.eqiad.wmnet |
[production] |
| 08:37 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[1276-1279].eqiad.wmnet |
[production] |
| 08:29 |
<effie> |
disable puppet on mediawiki hosts to merge 712920 |
[production] |
| 08:24 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw[1276-1279].eqiad.wmnet |
[production] |
| 08:24 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1456.eqiad.wmnet with reason: new setup |
[production] |
| 08:24 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1456.eqiad.wmnet with reason: new setup |
[production] |
| 08:21 |
<mutante> |
mw2383 - scap pull (still depooled because T286463 but alerts in Icinga since a while) |
[production] |
| 08:20 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1456.eqiad.wmnet with reason: REIMAGE |
[production] |
| 08:18 |
<zpapierski@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 08:18 |
<jelto@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw127[6-9].eqiad.wmnet |
[production] |
| 08:18 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1456.eqiad.wmnet with reason: REIMAGE |
[production] |
| 08:17 |
<jelto@cumin1001> |
conftool action : set/pooled=no; selector: name=mw127[6-9].eqiad.wmnet |
[production] |
| 08:15 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw[1276-1279].eqiad.wmnet with reason: decom old appservers in eqiad T280203 |
[production] |
| 08:15 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.downtime for 3:00:00 on mw[1276-1279].eqiad.wmnet with reason: decom old appservers in eqiad T280203 |
[production] |
| 08:06 |
<zpapierski@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 08:00 |
<jelto@cumin1001> |
conftool action : set/weight=30; selector: name=mw144[7-9].eqiad.wmnet |
[production] |
| 07:59 |
<mutante> |
mw1384 - start failed ferm service |
[production] |
| 07:59 |
<jelto@cumin1001> |
conftool action : set/weight=30; selector: name=mw1450.eqiad.wmnet |
[production] |
| 07:52 |
<mutante> |
mw1451 through mw1455 - fresh hardware pooled the first time as appservers |
[production] |
| 07:50 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw145[1-5].eqiad.wmnet |
[production] |
| 07:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw145[1-5].eqiad.wmnet |
[production] |