|
2021-10-08
§
|
| 04:32 |
<ryankemper> |
T292814 Beginning rolling restart of `cloudelastic`: `sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic restart" --nodes-per-run 1 --start-datetime 2021-10-08T03:53:49 --task-id T292814` on `ryankemper@cumin1001` tmux `elastic` |
[production] |
| 04:31 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814 |
[production] |
| 04:29 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 04:28 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across both test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 04:28 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 04:23 |
<ryankemper@deploy1002> |
Finished deploy [wdqs/wdqs@8f57a56]: 0.3.89 (duration: 08m 22s) |
[production] |
| 04:20 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814 |
[production] |
| 04:20 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814 |
[production] |
| 04:18 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) |
[production] |
| 04:17 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) |
[production] |
| 04:15 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.89` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 04:14 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@8f57a56]: 0.3.89 |
[production] |
| 04:14 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.89`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 03:58 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814 |
[production] |
| 03:58 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814 |
[production] |
| 02:04 |
<Krinkle> |
krinkle@deploy1002$ echo 'https://en.wikipedia.org/static/images/project-logos/jvwiktionary.png' | mwscript purgeList.php , ref T287425, T292810 |
[production] |
| 00:07 |
<tgr_> |
deploy window over |
[production] |
| 00:05 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.3/extensions/GrowthExperiments: Backport: [[gerrit:727498|Mentee overview: Make UncachedMenteeOverviewDataProvider::getBlocksForUsers faster (T290609)]] (duration: 00m 56s) |
[production] |
|
2021-10-07
§
|
| 23:43 |
<thcipriani@deploy1002> |
Synchronized wmf-config/logos.php: Config: [[gerrit:708065|Change Javanese Wiktionary logo (T287425)]] part 3/3 (duration: 00m 55s) |
[production] |
| 23:41 |
<thcipriani@deploy1002> |
Synchronized logos/config.yaml: Config: [[gerrit:708065|Change Javanese Wiktionary logo (T287425)]] part 2/3 (duration: 00m 55s) |
[production] |
| 23:40 |
<thcipriani@deploy1002> |
Synchronized static/images/project-logos: Config: [[gerrit:708065|Change Javanese Wiktionary logo (T287425)]] part 1/3 (duration: 00m 56s) |
[production] |
| 23:30 |
<thcipriani@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:704170|Adding and use wordmark in trwikiquote (T286133)]] Part 2/2 (duration: 00m 56s) |
[production] |
| 23:28 |
<thcipriani@deploy1002> |
Synchronized static/images/mobile/copyright/wikiquote-wordmark-tr.svg: Config: [[gerrit:704170|Adding and use wordmark in trwikiquote (T286133)]] Part 1/2 (duration: 00m 57s) |
[production] |
| 21:35 |
<urbanecm> |
Password reset for SUL User:LA2-bot (T292793) |
[production] |
| 20:43 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.3 |
[production] |
| 20:37 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.2 refs T281167 |
[production] |
| 20:35 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 20:35 |
<cmooney@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 20:23 |
<krinkle@deploy1002> |
Synchronized php-1.38.0-wmf.3/extensions/Gadgets/: I7c858b8c4bc (duration: 00m 56s) |
[production] |
| 20:01 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.3/extensions/Echo/: 8a7ff05ba28f302adb581bf430a868bb815b4ffd: Revert "Use namespaced CentralAuthSessionProvider" (duration: 00m 57s) |
[production] |
| 19:45 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.3/extensions/CentralAuth/: c01c2e4983bad8582ddd62aeb35ac9be852d493b: Revert "Namespace session providers" (duration: 00m 57s) |
[production] |
| 19:44 |
<urbanecm> |
Backporting https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/727489, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/727487 in an unsafe way -- exceptions at testwikis expected, wmf.3 is not deployed elsewhere, so this should be ok |
[production] |
| 19:37 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: Revert all wikis to 1.38.0-wmf.2 (T281167) |
[production] |
| 19:33 |
<brennen> |
1.38.0-wmf.3 train (T281167): variously blocked, rolling back to testwikis for safe deploy of backports |
[production] |
| 19:14 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: Revert group2 wikis to 1.38.0-wmf.2 |
[production] |
| 19:07 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.3 refs T281167 |
[production] |
| 19:03 |
<brennen> |
1.38.0-wmf.3 train (T281167): unblocked, rolling to all wikis |
[production] |
| 18:50 |
<urbanecm> |
[urbanecm@mwmaint1002 /srv/mediawiki/php]$ mwscript extensions/GrowthExperiments/maintenance/initWikiConfig.php --wiki=test2wiki |
[production] |
| 18:46 |
<sukhe> |
running authdns-update for T292537 |
[production] |
| 18:29 |
<urbanecm> |
Morning B&C window done |
[production] |
| 18:28 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 4a946c046ae17a520f8d3463a16b1435ceb4856c: Deploy Growth mentor dashboard to pilot wikis (T278920) (duration: 01m 04s) |
[production] |
| 18:23 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 87e300137c14451949fac12c3ec89319305a423e: Deploy Growth features to test2wiki (duration: 01m 03s) |
[production] |
| 18:21 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 87e300137c14451949fac12c3ec89319305a423e: Deploy Growth features to test2wiki (duration: 01m 04s) |
[production] |
| 18:20 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 31770f2b3660e7d7490c0a9ab66285c1f069732d: shwiki: Deploy Growth features to newcomers (T278240) (duration: 01m 04s) |
[production] |
| 18:15 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 33526dfed148068585289f5ac501feda72068fd9: Stream config changes for android_daily_stats schema (T286000) (duration: 01m 06s) |
[production] |
| 18:10 |
<ejegg> |
updated payments-wiki from 6d3560d083 to 030b11da1a |
[production] |
| 18:07 |
<arnoldokoth> |
gitlab2001 re-image complete (T283076) |
[production] |
| 17:30 |
<mutante> |
rebooting gitlab2001.wikimedia.org |
[production] |
| 16:56 |
<arnoldokoth> |
down timing gitlab2001 for re-imaging (T283076) |
[production] |
| 16:47 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2001.wikimedia.org with reason: reimage |
[production] |