2651-2700 of 10000 results (31ms)
2020-07-07 §
05:01 <marostegui> "Starting es failover from es1023 to es1024 - https://phabricator.wikimedia.org/T255755" [production]
01:05 <ejegg> turned on debug logging for Adyen SmashPig [production]
00:22 <cstone> civicrm revision changed from a48caf0f37 to d73ee2e73f [production]
2020-07-06 §
23:32 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable sidebar instrumentation on test wikipedia (duration: 00m 56s) [production]
23:32 <eileen> process-control config revision is 3fe6753e56 [production]
23:22 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Change some zh canonical namespaces. Don't index NS_USER on hywiki (duration: 00m 58s) [production]
22:58 <eileen> tools revision changed from e974147f27 to 73557b8038 [production]
22:14 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@65502b2]: 0.3.40 (duration: 18m 58s) [production]
21:55 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@65502b2]: 0.3.40 [production]
21:52 <hashar> Upgraded Jenkins on releases1002 and releases2002 # T256978 [production]
21:41 <mutante> upgrading jenkins on releases1001 and releases2001 (T256980) [production]
21:37 <mutante> importing jenkins 2.235.1 into APT repo for both stretch and buster T256980 [production]
20:08 <eileen> tools revision is e974147f27 [production]
19:41 <qchris> Enabling puppet on gerrit1002 again to catch up with puppetmaster. [production]
18:56 <addshore> backport / deploy window done [production]
18:55 <addshore@deploy1001> Synchronized wmf-config: [[gerrit:569263]] T241975 Wikibase: Remove config option wmgUseEntitySourceBasedFederation (duration: 00m 58s) [production]
18:54 <addshore@deploy1001> sync-file aborted: [[gerrit:569263]] (duration: 00m 00s) [production]
18:51 <addshore@deploy1001> Synchronized wmf-config/Wikibase.php: [[gerrit:608944]] T241975 Wikibase: stop using wmgUseEntitySourceBasedFederation (duration: 00m 56s) [production]
18:47 <addshore@deploy1001> Synchronized dblists/wikidataclient.dblist: T254315 Wikidata client wikis: Define entity sources configuration (take 2) [[gerrit:608839]] (duration: 00m 56s) [production]
18:45 <addshore@deploy1001> Synchronized wmf-config: T254315 Wikidata client wikis: Define entity sources configuration (take 2) [[gerrit:608839]] (duration: 00m 58s) [production]
18:38 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T256906 T256907 T256909 T254315 [[gerrit:569260]] Commons: Define entity sources configuration (duration: 00m 56s) [production]
18:30 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: adffbe6: Enable validation of new signatures (T248632) (duration: 00m 57s) [production]
18:24 <urbanecm@deploy1001> Synchronized wmf-config/abusefilter.php: 8878c60: Add `abusefilter-view` as a default right for the CU log user (T255506) (duration: 00m 55s) [production]
18:22 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 1398171: Add arbcom group to plwiki (T256572) (duration: 00m 56s) [production]
18:08 <andrew@deploy1001> Finished deploy [horizon/deploy@bb176c2]: update proxy UI to support multiple pre-set domains (duration: 03m 39s) [production]
18:04 <andrew@deploy1001> Started deploy [horizon/deploy@bb176c2]: update proxy UI to support multiple pre-set domains [production]
17:54 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate SearchSatisfaction from EventLogging to EventGate on all wikis - T249261 - take 2 (duration: 00m 56s) [production]
17:50 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate SearchSatisfaction from EventLogging to EventGate on all wikis - T249261 (duration: 00m 56s) [production]
16:09 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate SearchSatisfaction from EventLogging to EventGate on group1 - T249261 (duration: 00m 58s) [production]
15:02 <jynus> removing old snapshots for x1 on dbprov[12]002 [production]
14:50 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
14:46 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
14:44 <mholloway-shell@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
14:42 <moritzm> installing PHP 7.0 security updates [production]
14:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1074', diff saved to https://phabricator.wikimedia.org/P11753 and previous config saved to /var/cache/conftool/dbconfig/20200706-143754-marostegui.json [production]
14:36 <godog> reboot ms-be2025 for hw raid software upgrade - T257214 [production]
14:28 <godog> powercycle ms-be2025, no ssh available - T257214 [production]
14:14 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:09 <marostegui> Stop MySQL and poweroff db1079 T257216 [production]
14:08 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:02 <jynus@cumin1001> dbctl commit (dc=all): 'depool db1136 from main traffic as it is the only s7 api host right now', diff saved to https://phabricator.wikimedia.org/P11752 and previous config saved to /var/cache/conftool/dbconfig/20200706-140217-jynus.json [production]
13:56 <marostegui> Downtime and reboot db1079 after BBU crash [production]
13:54 <jynus@cumin1001> dbctl commit (dc=all): 'depool db1079', diff saved to https://phabricator.wikimedia.org/P11751 and previous config saved to /var/cache/conftool/dbconfig/20200706-135430-jynus.json [production]
13:30 <marostegui> Deploy schema change on s5 codfw master T253276 [production]
13:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Reduce es1024 weight in preparation for tomorrow's switchover T255755', diff saved to https://phabricator.wikimedia.org/P11750 and previous config saved to /var/cache/conftool/dbconfig/20200706-132634-marostegui.json [production]
13:03 <elukey> force umount/mount of /mnt/hdfs on an-airflow1001 to unblock dpkg checks (fuse misbehaving, all checks hanging) [production]
12:53 <elukey> kill hanging lsof processes on an-airflow to reduce cpu load [production]
12:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074', diff saved to https://phabricator.wikimedia.org/P11748 and previous config saved to /var/cache/conftool/dbconfig/20200706-124237-marostegui.json [production]
12:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1129', diff saved to https://phabricator.wikimedia.org/P11747 and previous config saved to /var/cache/conftool/dbconfig/20200706-124105-marostegui.json [production]
11:17 <Urbanecm> EU B&C window was done [production]