|
2021-08-17
ยง
|
| 17:41 |
<urbanecm> |
[urbanecm@mw2383 ~]$ scap pull # to clear an icinga alert |
[production] |
| 17:39 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.19/includes/: Backport: [[gerrit:713365|Revert "objectcache: make use of new `modtoken` field in SqlBagOStuff" (T288998)]] (duration: 01m 14s) |
[production] |
| 17:15 |
<bblack> |
authdns2001,dns[245]001: upgrade gdnsd package to 3.8.0-1~wmf1 (all authdns upgraded after this) |
[production] |
| 17:07 |
<mbsantos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 17:04 |
<mbsantos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 17:02 |
<mbsantos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 16:56 |
<brennen@deploy1002> |
Finished scap: testwikis wikis to 1.37.0-wmf.19 (duration: 38m 24s) |
[production] |
| 16:50 |
<bblack> |
dns1001: upgrade gdnsd package to 3.8.0-1~wmf1 |
[production] |
| 16:25 |
<bblack> |
dns3001: upgrade gdnsd package to 3.8.0-1~wmf1 |
[production] |
| 16:17 |
<brennen@deploy1002> |
Started scap: testwikis wikis to 1.37.0-wmf.19 |
[production] |
| 16:13 |
<brennen> |
1.37.0-wmf.19 train: running scap prep, branched at 79c9b9e61350b0edd1acccb5e717875ba64cf9c1 |
[production] |
| 16:08 |
<zpapierski@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 16:06 |
<zpapierski@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 16:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:55 |
<urbanecm> |
Deploy a security patch for T289064 |
[production] |
| 15:37 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 15:32 |
<zpapierski@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 15:23 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 15:06 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:40 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:37 |
<kormat@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Promote pc2013 to primary of pc3 T284825 (duration: 00m 58s) |
[production] |
| 14:25 |
<jynus> |
running a full testwiki media backup on a single thread, single worker T262668 |
[production] |
| 14:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:21 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:20 |
<kormat@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Promote pc2012 to primary of pc2 T284825 (duration: 00m 59s) |
[production] |
| 13:53 |
<jynus> |
rolling restart of minio on backup server |
[production] |
| 13:51 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 13:06 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 12:55 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 11:29 |
<phuedx@deploy1002> |
Synchronized php-1.37.0-wmf.18/extensions/SecurePoll/includes/Jobs/TallyElectionJob.php: Backport: [[gerrit:713361|tallyElectionJob: Catch and log exceptions (T288361)]] (duration: 00m 58s) |
[production] |
| 11:16 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17038 and previous config saved to /var/cache/conftool/dbconfig/20210817-111629-mvernon.json |
[production] |
| 11:15 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 11:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:01 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17037 and previous config saved to /var/cache/conftool/dbconfig/20210817-110125-mvernon.json |
[production] |
| 10:46 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17035 and previous config saved to /var/cache/conftool/dbconfig/20210817-104622-mvernon.json |
[production] |
| 10:31 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: buster reimage T288244', diff saved to https://phabricator.wikimedia.org/P17034 and previous config saved to /var/cache/conftool/dbconfig/20210817-103118-mvernon.json |
[production] |
| 10:07 |
<effie> |
enable puppet on mediawiki hosts |
[production] |
| 09:52 |
<mvernon@cumin2002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2121.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:50 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2121.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:20 |
<mvernon@cumin1001> |
dbctl commit (dc=all): 'db2121 depooling: reimage to buster T288244', diff saved to https://phabricator.wikimedia.org/P17033 and previous config saved to /var/cache/conftool/dbconfig/20210817-092045-mvernon.json |
[production] |
| 09:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1456.eqiad.wmnet |
[production] |
| 09:16 |
<Emperor> |
reimaging db2121 to buster T288244 |
[production] |
| 09:08 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw1456.eqiad.wmnet |
[production] |