|
2021-12-15
§
|
| 12:08 |
<moritzm> |
added ganeti2025 to codfw ganeti cluster T282603 |
[production] |
| 11:48 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance |
[production] |
| 11:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance |
[production] |
| 11:39 |
<_joe_> |
repooling mw1414 T297667 |
[production] |
| 11:36 |
<_joe_> |
upgrading php7.2 on mw1414, T297667 |
[production] |
| 11:35 |
<_joe_> |
uploading php 7.2 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf4 to buster T297667 |
[production] |
| 11:17 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2018.codfw.wmnet |
[production] |
| 11:11 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet |
[production] |
| 10:27 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:27 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:00 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 09:57 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 09:53 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . |
[production] |
| 09:43 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:42 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 09:28 |
<vgutierrez> |
pool cp4025 - T271421 |
[production] |
| 09:17 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2018.codfw.wmnet with OS buster |
[production] |
| 08:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2018.codfw.wmnet with OS buster |
[production] |
| 07:04 |
<marostegui> |
Enable full_crc32 on db2094 (s1, s3, s5 and s8) T287244 |
[production] |
| 05:48 |
<eileen> |
revision 1ede5365 -> d4cea6a9 civicrm |
[production] |
| 05:07 |
<eileen> |
revision d0ac9184 -> 1ede5365 civicrm |
[production] |
| 00:59 |
<dancy@deploy1002> |
Finished scap: testing (duration: 03m 38s) |
[production] |
| 00:55 |
<dancy@deploy1002> |
Started scap: testing |
[production] |
| 00:52 |
<dancy@deploy1002> |
Synchronized /: testing (duration: 00m 37s) |
[production] |
| 00:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:31 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.13/extensions/MediaSearch/resources/store/index.js: Backport: [[gerrit:747081|Remove multiple instance of VUEX initialization (T297690)]] (duration: 01m 04s) |
[production] |
| 00:29 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.13/extensions/MediaSearch/resources/components/SearchResults.vue: Backport: [[gerrit:747078|Don't attempt to scroll to a non-existing result]] (duration: 01m 05s) |
[production] |
| 00:28 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.13/includes/: Backport: [[gerrit:747079|Revert "Replace deprecated methods IContextSource::getWikiPage && IContextSource::canUseWikiPage" (T297744)]] (duration: 01m 12s) |
[production] |
| 00:26 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.12/includes/: Backport: [[gerrit:747080|Revert "Replace deprecated methods IContextSource::getWikiPage && IContextSource::canUseWikiPage" (T297744)]] (duration: 01m 11s) |
[production] |
| 00:01 |
<bblack> |
lvs1015: start pybal, back to normal |
[production] |
|
2021-12-14
§
|
| 23:49 |
<bblack> |
lvs1015 (internal services) - disabling pybal, will fail over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:44 |
<bblack> |
lvs1013 (text) restart pybal, back to normal |
[production] |
| 23:28 |
<bblack> |
lvs1013 (text) - disabling pybal, will fail over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:26 |
<bblack> |
lvs1014 (upload) restart pybal, back to normal |
[production] |
| 23:15 |
<bblack> |
lvs1014 (upload) - disabling pybal, will over traffic to lvs1020 (to test lvs1020 sanity) |
[production] |
| 23:10 |
<legoktm> |
deploying patch for T297416 |
[production] |
| 21:18 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.13 refs T293954 |
[production] |
| 21:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:09 |
<hashar@deploy1002> |
Finished scap: testwiki to php-1.38.0-wmf.13 and rebuild l10n cache (duration: 33m 47s) |
[production] |
| 20:43 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:35 |
<hashar@deploy1002> |
Started scap: testwiki to php-1.38.0-wmf.13 and rebuild l10n cache |
[production] |
| 20:34 |
<urbanecm> |
Manually rollback group0 to wmf.12 by running `sudo -u mwdeploy cp /srv/mediawiki-staging/wikiversions.json /srv/mediawiki/wikiversions.json && scap wikiversions-compile && cp /srv/mediawiki/wikiversions.php /srv/mediawiki-staging/wikiversions.php && scap sync-file --force wikiversions.php 'rollback group0'` |
[production] |
| 20:34 |
<hashar> |
Group 0 wikis are available again and still on 1.38.0-wmf.12 |
[production] |
| 20:31 |
<urbanecm@deploy1002> |
Synchronized wikiversions.php: rollback group0 (duration: 00m 41s) |
[production] |
| 20:28 |
<hashar> |
group0 wikis (eg mediawiki.org) are unavailable due to a deployment issue. We are working on it # T293954 |
[production] |
| 20:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |