|
2020-02-26
§
|
| 09:37 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers |
[production] |
| 09:34 |
<elukey> |
roll restart the Hadoop Analytcs workers for openjdk upgrades |
[production] |
| 09:32 |
<elukey> |
upgrade prometheus-mcrouter-exporter 0.1.0+git20200225-1 to all cumin alias all-mw-codfw |
[production] |
| 09:18 |
<gehel> |
restarting elasticsearch on cloudelastic for JVM upgrade |
[production] |
| 08:51 |
<elukey> |
upload prometheus-mcrouter-exporter 0.1.0+git20200225-1 to stretch-wikimedia |
[production] |
| 08:38 |
<elukey> |
upgrade prometheus-mcrouter-exporter on mwdebug1001 to test the new version |
[production] |
| 06:19 |
<marostegui> |
Stop MySQL and poweroff db1084 for BBU replacement - T245647 |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10530 and previous config saved to /var/cache/conftool/dbconfig/20200226-061710-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Restore es1017 (master) original weight (0) T243963', diff saved to https://phabricator.wikimedia.org/P10529 and previous config saved to /var/cache/conftool/dbconfig/20200226-061640-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1084 for BBU replacement - T245647', diff saved to https://phabricator.wikimedia.org/P10528 and previous config saved to /var/cache/conftool/dbconfig/20200226-060906-marostegui.json |
[production] |
| 05:41 |
<kart_> |
Updated cxserver to 2020-02-24-110149-production (T227183) |
[production] |
| 05:35 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:31 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:29 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 01:15 |
<ejegg> |
updated payments-wiki from c3ca3ad6a7 to bfae734204 |
[production] |
| 00:48 |
<eileen> |
civicrm revision changed from bec2d6ad9f to 62e62e107c, config revision is c0ef31e2fd |
[production] |
| 00:21 |
<James_F> |
Manually purged https://de.wikipedia.org/w/index.php?title=Hans-Werner_Sahm&action=history from mwmaint1002 |
[production] |
| 00:15 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 01m 03s) |
[production] |
| 00:15 |
<James_F> |
SWAT complete. |
[production] |
| 00:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T242381 Set Vector skin version defaults so they can be changed on Beta Cluster (duration: 01m 04s) |
[production] |
| 00:09 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 01m 03s) |
[production] |
| 00:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T245792 Enable password-reset-update on Wikivoyages and Wiktionaries (duration: 01m 04s) |
[production] |
| 00:08 |
<ebernhardson> |
resume writes from mediawiki to cloudelastic |
[production] |
|
2020-02-25
§
|
| 23:51 |
<XioNoX> |
cr2-esams> request chassis fpc slot 0 offline - T246009 |
[production] |
| 23:38 |
<ebernhardson> |
pause mediawiki writes to cloudelastic to let old gc on cloudelastic1001-chi recover |
[production] |
| 23:30 |
<mutante> |
notebook1004 - disk full once again (T232068) |
[production] |
| 23:28 |
<mutante> |
adding mw2366 through mw2376 to site |
[production] |
| 22:17 |
<jhuneidi@deploy1001> |
Synchronized php-1.35.0-wmf.21/includes/Defines.php: Update MW_VERSION to 1.35.0-wmf.21 (duration: 01m 04s) |
[production] |
| 22:17 |
<mutante> |
scandium restarting php7.2-fpm |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 21:29 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.21 refs T233869 |
[production] |
| 21:19 |
<jhuneidi@deploy1001> |
Finished scap: testwikis wikis to 1.35.0-wmf.21 refs T233869 (duration: 75m 21s) |
[production] |
| 20:42 |
<eileen> |
process-control config revision is c0ef31e2fd |
[production] |
| 20:32 |
<eileen> |
process-control config revision is e17d104c73 slow down delete deleted contacts |
[production] |
| 20:28 |
<tzatziki> |
reset password for ClioCJS |
[production] |
| 20:25 |
<tzatziki> |
changing email address for ClioCJS |
[production] |
| 20:25 |
<mutante> |
apt.wikimedia.org (current install* and new apt* roles) - going ECDSA-only and removing RSA certificate from nginx config - to support buster without having to maintain patched nginx for duplicate ssl_stapling_file directive - at the cost of slightly reduced back-compat on the public repo (T242602) |
[production] |
| 20:24 |
<mutante> |
apt.wikimedia.org (current install* and new apt* roles) - going ECDSA-only and removing RSA certificate from nginx config - to support buster without having to maintain patched nginx for duplicate ssl_stapling_file directive - at the cost of slightly reduced back-compat on the public repo (T224576) |
[production] |
| 20:18 |
<eileen> |
process-control config revision is e17d104c73 |
[production] |
| 20:04 |
<jhuneidi@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.21 refs T233869 |
[production] |
| 20:01 |
<jhuneidi@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.19 (duration: 14m 35s) |
[production] |
| 19:58 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 19:55 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 19:54 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 19:52 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 19:47 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 19:47 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 19:45 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 19:44 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |