|
2019-09-19
§
|
| 07:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:40 |
<moritzm> |
rebooting failoid1001 for kernel update |
[production] |
| 07:39 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:39 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:29 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:27 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give more logpager weight to db1089 T223151', diff saved to https://phabricator.wikimedia.org/P9131 and previous config saved to /var/cache/conftool/dbconfig/20190919-072234-marostegui.json |
[production] |
| 07:01 |
<moritzm> |
reimaging restbase2012 to stretch T224553 |
[production] |
| 06:18 |
<marostegui> |
Sanitize hiwikisource on db1124:3313 and db2094:3313 T219374 |
[production] |
| 06:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Temporarily pool db1089 into enwiki logpager T223151', diff saved to https://phabricator.wikimedia.org/P9130 and previous config saved to /var/cache/conftool/dbconfig/20190919-060440-marostegui.json |
[production] |
| 05:11 |
<marostegui> |
Stop MySQL on db2055 for decommission T233186 |
[production] |
| 05:11 |
<marostegui> |
Remove db2055 from tendril and zarcillo T233186 |
[production] |
|
2019-09-18
§
|
| 23:18 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/MobileFrontend/resources/dist/: T233260, 1667ed957a19067 (duration: 01m 04s) |
[production] |
| 22:58 |
<cmjohnson1> |
enabled asw2-c-eqiad interface xe-2/0/45 |
[production] |
| 22:40 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/resources/Resources.php: d6dadfdb0b237c918 (duration: 01m 03s) |
[production] |
| 22:37 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/AbuseFilter/includes/: T156095, ff44043efa59e9 (duration: 01m 05s) |
[production] |
| 22:13 |
<cmjohnson1> |
disabling asw2-c-eqiad xe-2/0/45 - cr1-eqiad to replace optic T233265 |
[production] |
| 21:54 |
<gilles> |
T233095 Purging all eswiki articles (both desktop and mobile this time) |
[production] |
| 21:53 |
<gilles@deploy1001> |
Synchronized php-1.34.0-wmf.22/maintenance/purgeList.php: T233095 Make purgeList.php use getCdnUrls() (duration: 01m 04s) |
[production] |
| 21:13 |
<XioNoX> |
enable damping on primary codfw-eqiad link - T196432 |
[production] |
| 21:09 |
<XioNoX> |
enable damping on codfw-ulsfo link - T196432 |
[production] |
| 20:50 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: No longer load InitialiseSettings at all in CommonSettings (duration: 01m 03s) |
[production] |
| 20:43 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Quick fix for wmfLoadInitialiseSettings() (duration: 01m 03s) |
[production] |
| 20:40 |
<jforrester@deploy1001> |
scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 20:23 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out call to InitialiseSettings.php (duration: 01m 04s) |
[production] |
| 20:18 |
<@> |
helmfile [EQIAD] Ran 'apply' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 20:18 |
<jforrester@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: Variant configuration: Drop suport for serialised PHP (duration: 01m 04s) |
[production] |
| 20:15 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Variant configuration: Never write to serialised PHP T223602 (duration: 01m 04s) |
[production] |
| 20:15 |
<@> |
helmfile [CODFW] Ran 'apply' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 20:11 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:07 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T208246 Enforce a 10-byte password for privileged users (duration: 01m 04s) |
[production] |
| 19:57 |
<urandom> |
decommissioning Cassandra, restbase2012-c -- T224553 |
[production] |
| 19:56 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 19:56 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:42 |
<gilles> |
T233095 Purging all pages on eswiki |
[production] |
| 19:27 |
<joal@deploy1001> |
Finished deploy [analytics/aqs/deploy@bc9dde1]: Regular deploy - analytics weekly train - Second retry after fix (duration: 03m 40s) |
[production] |
| 19:24 |
<mutante> |
ganeti1001 - deleting krypton.eqiad.wmnet - decom T231546 |
[production] |
| 19:23 |
<joal@deploy1001> |
Started deploy [analytics/aqs/deploy@bc9dde1]: Regular deploy - analytics weekly train - Second retry after fix |
[production] |
| 19:14 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.23 refs T220748 (duration: 01m 04s) |
[production] |
| 19:13 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.23 refs T220748 |
[production] |
| 19:10 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:08 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:07 |
<twentyafterfour> |
There appear to be no blockers on T220748 so I'll proceed with deploying 1.34.0-wmf.23 to group 1. |
[production] |
| 19:01 |
<joal@deploy1001> |
Finished deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train - Retry after fix (duration: 02m 12s) |
[production] |
| 18:59 |
<joal@deploy1001> |
Started deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train - Retry after fix |
[production] |
| 18:55 |
<joal@deploy1001> |
Finished deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train (duration: 01m 05s) |
[production] |
| 18:54 |
<joal@deploy1001> |
Started deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train |
[production] |
| 18:46 |
<XioNoX> |
remove `border-in4 term ddos-0906` from all routers |
[production] |
| 17:53 |
<Amir1> |
Creating hiwikisource is done |
[production] |
| 17:50 |
<urandom> |
decommissioning Cassandra, restbase2012-b -- T224553 |
[production] |