2501-2550 of 10000 results (71ms)
2019-09-19 §
07:29 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:27 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Give more logpager weight to db1089 T223151', diff saved to https://phabricator.wikimedia.org/P9131 and previous config saved to /var/cache/conftool/dbconfig/20190919-072234-marostegui.json [production]
07:01 <moritzm> reimaging restbase2012 to stretch T224553 [production]
06:18 <marostegui> Sanitize hiwikisource on db1124:3313 and db2094:3313 T219374 [production]
06:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Temporarily pool db1089 into enwiki logpager T223151', diff saved to https://phabricator.wikimedia.org/P9130 and previous config saved to /var/cache/conftool/dbconfig/20190919-060440-marostegui.json [production]
05:11 <marostegui> Stop MySQL on db2055 for decommission T233186 [production]
05:11 <marostegui> Remove db2055 from tendril and zarcillo T233186 [production]
2019-09-18 §
23:18 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.23/extensions/MobileFrontend/resources/dist/: T233260, 1667ed957a19067 (duration: 01m 04s) [production]
22:58 <cmjohnson1> enabled asw2-c-eqiad interface xe-2/0/45 [production]
22:40 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.23/resources/Resources.php: d6dadfdb0b237c918 (duration: 01m 03s) [production]
22:37 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.23/extensions/AbuseFilter/includes/: T156095, ff44043efa59e9 (duration: 01m 05s) [production]
22:13 <cmjohnson1> disabling asw2-c-eqiad xe-2/0/45 - cr1-eqiad to replace optic T233265 [production]
21:54 <gilles> T233095 Purging all eswiki articles (both desktop and mobile this time) [production]
21:53 <gilles@deploy1001> Synchronized php-1.34.0-wmf.22/maintenance/purgeList.php: T233095 Make purgeList.php use getCdnUrls() (duration: 01m 04s) [production]
21:13 <XioNoX> enable damping on primary codfw-eqiad link - T196432 [production]
21:09 <XioNoX> enable damping on codfw-ulsfo link - T196432 [production]
20:50 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: No longer load InitialiseSettings at all in CommonSettings (duration: 01m 03s) [production]
20:43 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Quick fix for wmfLoadInitialiseSettings() (duration: 01m 03s) [production]
20:40 <jforrester@deploy1001> scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
20:23 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out call to InitialiseSettings.php (duration: 01m 04s) [production]
20:18 <@> helmfile [EQIAD] Ran 'apply' command on namespace 'sessionstore' for release 'production' . [production]
20:18 <jforrester@deploy1001> Synchronized multiversion/MWConfigCacheGenerator.php: Variant configuration: Drop suport for serialised PHP (duration: 01m 04s) [production]
20:15 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Variant configuration: Never write to serialised PHP T223602 (duration: 01m 04s) [production]
20:15 <@> helmfile [CODFW] Ran 'apply' command on namespace 'sessionstore' for release 'production' . [production]
20:11 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
20:07 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T208246 Enforce a 10-byte password for privileged users (duration: 01m 04s) [production]
19:57 <urandom> decommissioning Cassandra, restbase2012-c -- T224553 [production]
19:56 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
19:56 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:42 <gilles> T233095 Purging all pages on eswiki [production]
19:27 <joal@deploy1001> Finished deploy [analytics/aqs/deploy@bc9dde1]: Regular deploy - analytics weekly train - Second retry after fix (duration: 03m 40s) [production]
19:24 <mutante> ganeti1001 - deleting krypton.eqiad.wmnet - decom T231546 [production]
19:23 <joal@deploy1001> Started deploy [analytics/aqs/deploy@bc9dde1]: Regular deploy - analytics weekly train - Second retry after fix [production]
19:14 <twentyafterfour@deploy1001> Synchronized php: group1 wikis to 1.34.0-wmf.23 refs T220748 (duration: 01m 04s) [production]
19:13 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.23 refs T220748 [production]
19:10 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:08 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:07 <twentyafterfour> There appear to be no blockers on T220748 so I'll proceed with deploying 1.34.0-wmf.23 to group 1. [production]
19:01 <joal@deploy1001> Finished deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train - Retry after fix (duration: 02m 12s) [production]
18:59 <joal@deploy1001> Started deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train - Retry after fix [production]
18:55 <joal@deploy1001> Finished deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train (duration: 01m 05s) [production]
18:54 <joal@deploy1001> Started deploy [analytics/aqs/deploy@5b011d1]: Regular deploy - analytics weekly train [production]
18:46 <XioNoX> remove `border-in4 term ddos-0906` from all routers [production]
17:53 <Amir1> Creating hiwikisource is done [production]
17:50 <urandom> decommissioning Cassandra, restbase2012-b -- T224553 [production]
17:48 <ladsgroup@deploy1001> Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 32s) [production]
17:45 <ladsgroup@deploy1001> Synchronized static/images/project-logos/: Add hiwikisource logos (T218155) (duration: 01m 04s) [production]
17:43 <ladsgroup@deploy1001> Synchronized wmf-config/VariantSettings.php: Add hiwikisource (T218155) (duration: 01m 05s) [production]
17:40 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add hiwikisource (T218155) (duration: 01m 04s) [production]