|
2020-03-17
ยง
|
| 18:37 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 18:35 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw123[8-9].eqiad.wmnet |
[production] |
| 18:35 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw124[0-3].eqiad.wmnet |
[production] |
| 18:29 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 18:01 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@b6bff94]: Update mobileapps to 3c73ca3 (duration: 06m 06s) |
[production] |
| 18:00 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 17:58 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 17:56 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.23/languages/LanguageConverter.php: [[gerrit:580361|languages: Don't assume in LanguageConverter (T235360)]] (duration: 01m 07s) |
[production] |
| 17:55 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@b6bff94]: Update mobileapps to 3c73ca3 |
[production] |
| 17:55 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 17:53 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw124[0-3].eqiad.wmnet |
[production] |
| 17:53 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw123[89].eqiad.wmnet |
[production] |
| 17:52 |
<Amir1> |
warming up cache for Q70M to Q80M for new term store on db1111, db1126, db1104, db1092 (T219123) |
[production] |
| 17:46 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php: [[gerrit:580352|Do not lock rows when there's no term returned (T247553 T246898)]] (duration: 01m 07s) |
[production] |
| 17:42 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 17:40 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 17:37 |
<ejegg> |
updated payments-wiki from 86ce0361f9 to 72856949a1 |
[production] |
| 17:30 |
<bearND> |
mobileapps deploy failed on canary, rolled back |
[production] |
| 17:29 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@266e6da]: Update mobileapps to 6370784 (duration: 04m 00s) |
[production] |
| 17:25 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@266e6da]: Update mobileapps to 6370784 |
[production] |
| 17:24 |
<elukey@deploy1001> |
Finished deploy [analytics/superset/deploy@3f3ddcb]: Upgrade PyHive to 0.6.2 (duration: 00m 43s) |
[production] |
| 17:24 |
<elukey@deploy1001> |
Started deploy [analytics/superset/deploy@3f3ddcb]: Upgrade PyHive to 0.6.2 |
[production] |
| 17:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1280.eqiad.wmnet |
[production] |
| 17:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1280.eqiad.wmnet |
[production] |
| 17:10 |
<jynus> |
purging some old rows on pc1010 on a screen to earn some time T247788 |
[production] |
| 16:56 |
<mutante> |
mw1280 - scap pull - had ancient mw version due to downtime |
[production] |
| 16:46 |
<mutante> |
mw1280 back after long downtime due to broken RAM, added back into puppet (T240187) |
[production] |
| 16:36 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:36 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:56 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Reverting All wikis to 1.35.0-wmf.23 |
[production] |
| 15:52 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 15:52 |
<brennen@deploy1001> |
sync-wikiversions aborted: All wikis to 1.35.0-wmf.23 (duration: 05m 16s) |
[production] |
| 15:51 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 15:50 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 15:44 |
<brennen@deploy1001> |
sync-wikiversions aborted: All wikis to 1.35.0-wmf.23 (duration: 03m 49s) |
[production] |
| 15:36 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 15:36 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 15:23 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 15:11 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 15:01 |
<hashar> |
scap prep 1.35.0-wmf.24 and applying security patches # T233872 |
[production] |
| 15:00 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:57 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:44 |
<dcausse> |
wdqs1010 (test server) is running a data-reload cookbook (and is probably taking longer than the expected downtime) |
[production] |
| 14:38 |
<hashar> |
mediawiki/core git push 68bc9300dc:wmf/1.35.0-wmf.24 to catch up with a change that got merged while branch is being cut # T233872 |
[production] |
| 14:29 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Set up read new term store up to Q70M (T219123)]], take II (duration: 01m 04s) |
[production] |
| 14:28 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Set up read new term store up to Q70M (T219123)]] (duration: 01m 10s) |
[production] |
| 14:23 |
<marostegui> |
Stop mysql and restart pc1008 T247787 |
[production] |
| 14:23 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |