|
2021-12-15
§
|
| 11:11 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet |
[production] |
| 10:27 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:27 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:00 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 09:57 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 09:53 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . |
[production] |
| 09:43 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:42 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 09:28 |
<vgutierrez> |
pool cp4025 - T271421 |
[production] |
| 09:17 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2018.codfw.wmnet with OS buster |
[production] |
| 08:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2018.codfw.wmnet with OS buster |
[production] |
| 07:04 |
<marostegui> |
Enable full_crc32 on db2094 (s1, s3, s5 and s8) T287244 |
[production] |
| 05:48 |
<eileen> |
revision 1ede5365 -> d4cea6a9 civicrm |
[production] |
| 05:07 |
<eileen> |
revision d0ac9184 -> 1ede5365 civicrm |
[production] |
| 00:59 |
<dancy@deploy1002> |
Finished scap: testing (duration: 03m 38s) |
[production] |
| 00:55 |
<dancy@deploy1002> |
Started scap: testing |
[production] |
| 00:52 |
<dancy@deploy1002> |
Synchronized /: testing (duration: 00m 37s) |
[production] |
| 00:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:31 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.13/extensions/MediaSearch/resources/store/index.js: Backport: [[gerrit:747081|Remove multiple instance of VUEX initialization (T297690)]] (duration: 01m 04s) |
[production] |
| 00:29 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.13/extensions/MediaSearch/resources/components/SearchResults.vue: Backport: [[gerrit:747078|Don't attempt to scroll to a non-existing result]] (duration: 01m 05s) |
[production] |
| 00:28 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.13/includes/: Backport: [[gerrit:747079|Revert "Replace deprecated methods IContextSource::getWikiPage && IContextSource::canUseWikiPage" (T297744)]] (duration: 01m 12s) |
[production] |
| 00:26 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.12/includes/: Backport: [[gerrit:747080|Revert "Replace deprecated methods IContextSource::getWikiPage && IContextSource::canUseWikiPage" (T297744)]] (duration: 01m 11s) |
[production] |
| 00:01 |
<bblack> |
lvs1015: start pybal, back to normal |
[production] |
|
2021-12-14
§
|
| 23:49 |
<bblack> |
lvs1015 (internal services) - disabling pybal, will fail over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:44 |
<bblack> |
lvs1013 (text) restart pybal, back to normal |
[production] |
| 23:28 |
<bblack> |
lvs1013 (text) - disabling pybal, will fail over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:26 |
<bblack> |
lvs1014 (upload) restart pybal, back to normal |
[production] |
| 23:15 |
<bblack> |
lvs1014 (upload) - disabling pybal, will over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:10 |
<legoktm> |
deploying patch for T297416 |
[production] |
| 21:18 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.13 refs T293954 |
[production] |
| 21:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:09 |
<hashar@deploy1002> |
Finished scap: testwiki to php-1.38.0-wmf.13 and rebuild l10n cache (duration: 33m 47s) |
[production] |
| 20:43 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:35 |
<hashar@deploy1002> |
Started scap: testwiki to php-1.38.0-wmf.13 and rebuild l10n cache |
[production] |
| 20:34 |
<urbanecm> |
Manually rollback group0 to wmf.12 by running `sudo -u mwdeploy cp /srv/mediawiki-staging/wikiversions.json /srv/mediawiki/wikiversions.json && scap wikiversions-compile && cp /srv/mediawiki/wikiversions.php /srv/mediawiki-staging/wikiversions.php && scap sync-file --force wikiversions.php 'rollback group0'` |
[production] |
| 20:34 |
<hashar> |
Group 0 wikis are available again and still on 1.38.0-wmf.12 |
[production] |
| 20:31 |
<urbanecm@deploy1002> |
Synchronized wikiversions.php: rollback group0 (duration: 00m 41s) |
[production] |
| 20:28 |
<hashar> |
group0 wikis (eg mediawiki.org) are unavailable due to a deployment issue. We are working on it # T293954 |
[production] |
| 20:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:16 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.13 refs T293954 |
[production] |
| 20:15 |
<eileen> |
a88cd178 -> d0ac9184 |
[production] |
| 20:02 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:58 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: e127f4c6459cd9bc708b35a75c1f272b96fc3211: zhwiki: Promote Growth features out of dark mode (T287884) (duration: 00m 57s) |
[production] |
| 19:54 |
<urbanecm> |
UTC evening B&C window done |
[production] |
| 19:53 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.12/skins/Vector/resources/skins.vector.es6/AB.js: 62e84e7467c1765986cd1f80b466b8cacc6d91f6: Prevent A/B test enrollment hook from firing for unsampled (T297662) (duration: 00m 56s) |
[production] |