|
2019-09-19
§
|
| 14:18 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:16 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@44f4c79]: Remove the TID suffix in the ETag, if present - T230272 |
[production] |
| 13:54 |
<mholloway-shell@deploy1001> |
Finished deploy [recommendation-api/deploy@c8abb0f]: Article recommendation API: replace WDQS with MW API (T216750) (duration: 03m 06s) |
[production] |
| 13:51 |
<mholloway-shell@deploy1001> |
Started deploy [recommendation-api/deploy@c8abb0f]: Article recommendation API: replace WDQS with MW API (T216750) |
[production] |
| 13:43 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/Translate: T233308 (duration: 01m 07s) |
[production] |
| 13:14 |
<moritzm> |
powercycling mw1300 |
[production] |
| 13:12 |
<mobrovac> |
bootstrap restbase2012-b -- T224553 |
[production] |
| 13:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1089 into contributions service T223151', diff saved to https://phabricator.wikimedia.org/P9133 and previous config saved to /var/cache/conftool/dbconfig/20190919-130848-marostegui.json |
[production] |
| 13:01 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@7f4b7f7]: Start using RESTBase built on Stretch - T224553 (duration: 21m 38s) |
[production] |
| 12:39 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@7f4b7f7]: Start using RESTBase built on Stretch - T224553 |
[production] |
| 12:36 |
<@> |
helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' . |
[production] |
| 11:48 |
<mobrovac> |
bootstrap restbase2012-a -- T224553 |
[production] |
| 11:32 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:26 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: 199a05c: Add new throttle rule for Czech wiki course (T233199) (duration: 01m 01s) |
[production] |
| 11:23 |
<urbanecm@deploy1001> |
Synchronized wmf-config/VariantSettings.php: SWAT: eab7c6a: c80f026: GrowthExperiments: GrowthExperiments: Enable Special:Homepage for euwiki, GrowthExperiments: Enable help panel for euwiki (T233066, T233065) (duration: 01m 05s) |
[production] |
| 09:54 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.22/extensions/CheckUser: security T207094 (duration: 01m 02s) |
[production] |
| 09:53 |
<urbanecm@deploy1001> |
sync-file aborted: security T207094 (duration: 00m 28s) |
[production] |
| 09:51 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/CheckUser: security T207094 (duration: 01m 05s) |
[production] |
| 09:22 |
<godog> |
power back on ms-be1027, found with power off |
[production] |
| 08:31 |
<urbanecm@deploy1001> |
Synchronized wmf-config/abusefilter.php: 393441b: Change configuration of AbuseFilter extension for enwikisource (T231750) (duration: 01m 04s) |
[production] |
| 08:22 |
<jynus@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:21 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/CheckUser/: revert T207094 (duration: 01m 04s) |
[production] |
| 08:20 |
<jynus@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:14 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/CheckUser/: security T207094 (duration: 01m 06s) |
[production] |
| 08:11 |
<marostegui> |
Rename tables on db1133:labspuppet T233281 |
[production] |
| 07:48 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:40 |
<moritzm> |
rebooting failoid1001 for kernel update |
[production] |
| 07:39 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:39 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:29 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:27 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give more logpager weight to db1089 T223151', diff saved to https://phabricator.wikimedia.org/P9131 and previous config saved to /var/cache/conftool/dbconfig/20190919-072234-marostegui.json |
[production] |
| 07:01 |
<moritzm> |
reimaging restbase2012 to stretch T224553 |
[production] |
| 06:18 |
<marostegui> |
Sanitize hiwikisource on db1124:3313 and db2094:3313 T219374 |
[production] |
| 06:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Temporarily pool db1089 into enwiki logpager T223151', diff saved to https://phabricator.wikimedia.org/P9130 and previous config saved to /var/cache/conftool/dbconfig/20190919-060440-marostegui.json |
[production] |
| 05:11 |
<marostegui> |
Stop MySQL on db2055 for decommission T233186 |
[production] |
| 05:11 |
<marostegui> |
Remove db2055 from tendril and zarcillo T233186 |
[production] |
|
2019-09-18
§
|
| 23:18 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/MobileFrontend/resources/dist/: T233260, 1667ed957a19067 (duration: 01m 04s) |
[production] |
| 22:58 |
<cmjohnson1> |
enabled asw2-c-eqiad interface xe-2/0/45 |
[production] |
| 22:40 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/resources/Resources.php: d6dadfdb0b237c918 (duration: 01m 03s) |
[production] |
| 22:37 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/AbuseFilter/includes/: T156095, ff44043efa59e9 (duration: 01m 05s) |
[production] |
| 22:13 |
<cmjohnson1> |
disabling asw2-c-eqiad xe-2/0/45 - cr1-eqiad to replace optic T233265 |
[production] |
| 21:54 |
<gilles> |
T233095 Purging all eswiki articles (both desktop and mobile this time) |
[production] |
| 21:53 |
<gilles@deploy1001> |
Synchronized php-1.34.0-wmf.22/maintenance/purgeList.php: T233095 Make purgeList.php use getCdnUrls() (duration: 01m 04s) |
[production] |
| 21:13 |
<XioNoX> |
enable damping on primary codfw-eqiad link - T196432 |
[production] |
| 21:09 |
<XioNoX> |
enable damping on codfw-ulsfo link - T196432 |
[production] |
| 20:50 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: No longer load InitialiseSettings at all in CommonSettings (duration: 01m 03s) |
[production] |
| 20:43 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Quick fix for wmfLoadInitialiseSettings() (duration: 01m 03s) |
[production] |
| 20:40 |
<jforrester@deploy1001> |
scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |