|
2020-02-26
§
|
| 14:11 |
<volans@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:05 |
<gehel> |
restart of elasticsearch on cloudelastic for JVM upgrade completed |
[production] |
| 14:03 |
<XioNoX> |
deactivate BGP to AS23930 on cr1-eqsin, will re-enable when their technical issues are fixed and they notify us |
[production] |
| 14:00 |
<elukey> |
run apt-get clean on notebook1004 to free some space - T224682 |
[production] |
| 13:45 |
<XioNoX> |
ganeti2001:~$ sudo gnt-instance shutdown apt2001.wikimedia.org - T224576 |
[production] |
| 12:26 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:26 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:24 |
<kartik@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit|416973|ContentTranslation: Set cookieDomain for Production]] (duration: 01m 04s) |
[production] |
| 12:11 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|574469|Enable CX out of beta in eu, sw, and ta Wikipedias (T245446, T245447, T245448)]] take II (duration: 01m 05s) |
[production] |
| 12:10 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|574469|Enable CX out of beta in eu, sw, and ta Wikipedias (T245446, T245447, T245448)]] (duration: 01m 15s) |
[production] |
| 12:05 |
<volans> |
uploaded spicerack_0.0.31-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 11:45 |
<jbond42> |
changing uid/gid of reprepro effects release[12]001/install[12]002 |
[production] |
| 11:05 |
<moritzm> |
rolling out remaining PHP 7.0 security updates |
[production] |
| 10:57 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) |
[production] |
| 10:52 |
<moritzm> |
installing clamav security updates on mendelevium (ticket.wikimedia.org |
[production] |
| 10:03 |
<elukey> |
upgrade prometheus-mcrouter-exporter 0.1.0+git20200225-1 to all cumin alias parsoid/deployment-servers/mw-maintenance |
[production] |
| 09:54 |
<elukey> |
upgrade prometheus-mcrouter-exporter 0.1.0+git20200225-1 to all cumin alias all-mw-eqiad |
[production] |
| 09:37 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers |
[production] |
| 09:34 |
<elukey> |
roll restart the Hadoop Analytcs workers for openjdk upgrades |
[production] |
| 09:32 |
<elukey> |
upgrade prometheus-mcrouter-exporter 0.1.0+git20200225-1 to all cumin alias all-mw-codfw |
[production] |
| 09:18 |
<gehel> |
restarting elasticsearch on cloudelastic for JVM upgrade |
[production] |
| 08:51 |
<elukey> |
upload prometheus-mcrouter-exporter 0.1.0+git20200225-1 to stretch-wikimedia |
[production] |
| 08:38 |
<elukey> |
upgrade prometheus-mcrouter-exporter on mwdebug1001 to test the new version |
[production] |
| 06:19 |
<marostegui> |
Stop MySQL and poweroff db1084 for BBU replacement - T245647 |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10530 and previous config saved to /var/cache/conftool/dbconfig/20200226-061710-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Restore es1017 (master) original weight (0) T243963', diff saved to https://phabricator.wikimedia.org/P10529 and previous config saved to /var/cache/conftool/dbconfig/20200226-061640-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1084 for BBU replacement - T245647', diff saved to https://phabricator.wikimedia.org/P10528 and previous config saved to /var/cache/conftool/dbconfig/20200226-060906-marostegui.json |
[production] |
| 05:41 |
<kart_> |
Updated cxserver to 2020-02-24-110149-production (T227183) |
[production] |
| 05:35 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:31 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:29 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 01:15 |
<ejegg> |
updated payments-wiki from c3ca3ad6a7 to bfae734204 |
[production] |
| 00:48 |
<eileen> |
civicrm revision changed from bec2d6ad9f to 62e62e107c, config revision is c0ef31e2fd |
[production] |
| 00:21 |
<James_F> |
Manually purged https://de.wikipedia.org/w/index.php?title=Hans-Werner_Sahm&action=history from mwmaint1002 |
[production] |
| 00:15 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 01m 03s) |
[production] |
| 00:15 |
<James_F> |
SWAT complete. |
[production] |
| 00:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T242381 Set Vector skin version defaults so they can be changed on Beta Cluster (duration: 01m 04s) |
[production] |
| 00:09 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Bonus sync for cache clearance (duration: 01m 03s) |
[production] |
| 00:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T245792 Enable password-reset-update on Wikivoyages and Wiktionaries (duration: 01m 04s) |
[production] |
| 00:08 |
<ebernhardson> |
resume writes from mediawiki to cloudelastic |
[production] |
|
2020-02-25
§
|
| 23:51 |
<XioNoX> |
cr2-esams> request chassis fpc slot 0 offline - T246009 |
[production] |
| 23:38 |
<ebernhardson> |
pause mediawiki writes to cloudelastic to let old gc on cloudelastic1001-chi recover |
[production] |
| 23:30 |
<mutante> |
notebook1004 - disk full once again (T232068) |
[production] |
| 23:28 |
<mutante> |
adding mw2366 through mw2376 to site |
[production] |
| 22:17 |
<jhuneidi@deploy1001> |
Synchronized php-1.35.0-wmf.21/includes/Defines.php: Update MW_VERSION to 1.35.0-wmf.21 (duration: 01m 04s) |
[production] |
| 22:17 |
<mutante> |
scandium restarting php7.2-fpm |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 22:15 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 21:29 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.21 refs T233869 |
[production] |
| 21:19 |
<jhuneidi@deploy1001> |
Finished scap: testwikis wikis to 1.35.0-wmf.21 refs T233869 (duration: 75m 21s) |
[production] |