|
2021-12-14
§
|
| 23:49 |
<bblack> |
lvs1015 (internal services) - disabling pybal, will fail over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:44 |
<bblack> |
lvs1013 (text) restart pybal, back to normal |
[production] |
| 23:28 |
<bblack> |
lvs1013 (text) - disabling pybal, will fail over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:26 |
<bblack> |
lvs1014 (upload) restart pybal, back to normal |
[production] |
| 23:15 |
<bblack> |
lvs1014 (upload) - disabling pybal, will over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:10 |
<legoktm> |
deploying patch for T297416 |
[production] |
| 21:18 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.13 refs T293954 |
[production] |
| 21:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:09 |
<hashar@deploy1002> |
Finished scap: testwiki to php-1.38.0-wmf.13 and rebuild l10n cache (duration: 33m 47s) |
[production] |
| 20:43 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:35 |
<hashar@deploy1002> |
Started scap: testwiki to php-1.38.0-wmf.13 and rebuild l10n cache |
[production] |
| 20:34 |
<urbanecm> |
Manually rollback group0 to wmf.12 by running `sudo -u mwdeploy cp /srv/mediawiki-staging/wikiversions.json /srv/mediawiki/wikiversions.json && scap wikiversions-compile && cp /srv/mediawiki/wikiversions.php /srv/mediawiki-staging/wikiversions.php && scap sync-file --force wikiversions.php 'rollback group0'` |
[production] |
| 20:34 |
<hashar> |
Group 0 wikis are available again and still on 1.38.0-wmf.12 |
[production] |
| 20:31 |
<urbanecm@deploy1002> |
Synchronized wikiversions.php: rollback group0 (duration: 00m 41s) |
[production] |
| 20:28 |
<hashar> |
group0 wikis (eg mediawiki.org) are unavailable due to a deployment issue. We are working on it # T293954 |
[production] |
| 20:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:16 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.13 refs T293954 |
[production] |
| 20:15 |
<eileen> |
a88cd178 -> d0ac9184 |
[production] |
| 20:02 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:58 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: e127f4c6459cd9bc708b35a75c1f272b96fc3211: zhwiki: Promote Growth features out of dark mode (T287884) (duration: 00m 57s) |
[production] |
| 19:54 |
<urbanecm> |
UTC evening B&C window done |
[production] |
| 19:53 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.12/skins/Vector/resources/skins.vector.es6/AB.js: 62e84e7467c1765986cd1f80b466b8cacc6d91f6: Prevent A/B test enrollment hook from firing for unsampled (T297662) (duration: 00m 56s) |
[production] |
| 19:51 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 40f0cff8da7c4484e1fe93b9d649fd03f462e434: VE on zh.wiki: Enable single-edit-tab mode, and other config like en.wiki (T296269) (duration: 00m 57s) |
[production] |
| 19:47 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 7f4ae4cc678aa64b0795be7bc4c9a6f1ba4c1929: kartographer: Enable tegola on jawiki (T280767) (duration: 00m 58s) |
[production] |
| 19:29 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:19 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic rolling restart - ryankemper@cumin1001 - T297468 |
[production] |
| 19:18 |
<bblack> |
lvs1020 - rebooting on new config |
[production] |
| 19:17 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:16 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:08 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@92c63c9] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@92c63c9] (duration: 06m 54s) |
[production] |
| 19:01 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@92c63c9] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@92c63c9] |
[production] |
| 19:01 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@92c63c9] (thin): Regular analytics weekly train THIN [analytics/refinery@92c63c9] (duration: 00m 07s) |
[production] |
| 19:01 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@92c63c9] (thin): Regular analytics weekly train THIN [analytics/refinery@92c63c9] |
[production] |
| 18:59 |
<bblack> |
lvs1020: running puppet agent with lvs role + config for first time |
[production] |
| 18:58 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@92c63c9]: Regular analytics weekly train [analytics/refinery@92c63c9] (duration: 19m 49s) |
[production] |
| 18:40 |
<bblack> |
lvs1016: puppet agent disabled, pybal stopped |
[production] |
| 18:39 |
<bblack> |
lvs1016: downtimed for attempt at moving its role to lvs1020 (expect a few minor related alerts, such as BGP ones for eqiad routers) |
[production] |
| 18:38 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@92c63c9]: Regular analytics weekly train [analytics/refinery@92c63c9] |
[production] |
| 18:34 |
<majavah> |
deployed updated patch for T297322 |
[production] |
| 18:28 |
<ryankemper> |
T297468 [Elastic] `ryankemper@relforge1003:~$ sudo systemctl restart elasticsearch_6@relforge-eqiad.service elasticsearch_6@relforge-eqiad-small-alpha.service logstash.service` |
[production] |
| 18:25 |
<otto@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw |
[production] |
| 18:25 |
<ottomata> |
repooling eventgate-main discovery to include codfw - T296699 - confctl --object-type discovery select 'dnsdisc=eventgate-main,name=codfw' set/pooled=true |
[production] |