451-500 of 10000 results (27ms)
2020-10-15 §
02:13 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0) [production]
00:36 <ryankemper> Beginning rolling upgrade for cirrussearch `eqiad`. Cookbook will restart elasticsearch on 36 nodes total, 3 nodes at a time [production]
00:36 <eileen> tools revision changed from d4e08c52de to a2a91d6c6a [production]
00:35 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
00:24 <twentyafterfour> phabricator update was uneventful [production]
00:13 <twentyafterfour> updating phabricator [production]
2020-10-14 §
23:35 <foks> Removing one further file for legal compliance [production]
23:28 <foks> Removing nine files for legal compliance [production]
23:11 <ebernhardson> Syncronized wmf-config/InitialiseSettings.php to sync reduction of cirrus morelike query cache from 3 back to 1 day [production]
23:08 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 01m 04s) [production]
23:00 <dwisehaupt> all payments hosts in eqiad are now running the REL1_35 code. [production]
22:41 <ebernhardson@deploy1001> Finished deploy [search/mjolnir/deploy@9ce273f]: bulk_daemon: revert of streaming gzip decompression (duration: 02m 25s) [production]
22:38 <ebernhardson@deploy1001> Started deploy [search/mjolnir/deploy@9ce273f]: bulk_daemon: revert of streaming gzip decompression [production]
22:13 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.36.0-wmf.13 (duration: 01m 03s) [production]
22:12 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.13 [production]
22:08 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@04548dd]: spark: centralize reading/writing to hive (duration: 03m 44s) [production]
22:04 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@04548dd]: spark: centralize reading/writing to hive [production]
22:01 <thcipriani@deploy1001> Synchronized php-1.36.0-wmf.13/extensions/NavigationTiming: BACON: [[gerrit:634002|Make attribution source logic more defensive]] T263599 (duration: 01m 05s) [production]
21:51 <dpifke@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enabling image preconnect in group0 (T123582) (duration: 01m 03s) [production]
21:33 <thcipriani@deploy1001> Synchronized php-1.36.0-wmf.13/skins/Vector/resources/skins.vector.styles/Menu.less: BACON: [[gerrit:634086|Stylesheet needs to be compatible with cached HTML]] T265543 (duration: 01m 07s) [production]
20:39 <marxarelli> group1 rolled back to 1.36.0-wmf.11 due to malformed html in nav. task incoming (cc: T263179) [production]
20:37 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: Revert group1 wikis to 1.36.0-wmf.11 [production]
20:32 <marxarelli> rolling back group1 due to malformed html in nav menu [production]
19:46 <marxarelli> 1.36.0-wmf.13 promoted to group1. no new or concerning errors or changes in error rates (T263179) [production]
19:39 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.36.0-wmf.13 (duration: 01m 03s) [production]
19:38 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.13 [production]
19:33 <mutante> mx1001/mx2001 - temp. disabled puppet, live hacking urgent alias change since private repo needs to be fixed [production]
19:14 <mutante> depooling 5 of the older parsoid servers in codfw [production]
19:14 <dzahn@cumin1001> conftool action : set/pooled=no; selector: dc=codfw,name=wtp200[1-5].codfw.wmnet [production]
18:28 <Urbanecm> wikiadmin@10.192.0.6(wikidatawiki)> DELETE FROM watchlist WHERE wl_user=104889; # T265347 [production]
18:14 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: d6a56bb7fb762c53db5965f2698a93db2433d33d: Add rollbacker right on uzwiki (T265509) (duration: 01m 04s) [production]
18:10 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 0da89998e4e380f3ebe527a42a47dc66c49ee4d2: Add spamblacklistlog as a default right for the CU log user (T239288) (duration: 01m 05s) [production]
16:12 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) [production]
15:59 <elukey> drain + reboot an-worker1100 to pick up GPU settings - T255138 [production]
15:58 <elukey@cumin1001> START - Cookbook sre.hadoop.reboot-workers [production]
15:55 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) [production]
15:29 <elukey> drain + reboot an-worker110[1,2] to pick up GPU settings - T255138 [production]
15:28 <elukey@cumin1001> START - Cookbook sre.hadoop.reboot-workers [production]
15:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) [production]
15:24 <jayme> enabled and ran puppet on deploy1001 - T260917 [production]
14:56 <elukey> drain + reboot an-worker109[8,9] to pick up GPU settings - T255138 [production]
14:55 <elukey@cumin1001> START - Cookbook sre.hadoop.reboot-workers [production]
14:12 <jayme> disable-puppet on deploy1001 to test a change in hemlfile puppet on deploy2001 only - T260917 [production]
14:01 <akosiaris> push a 6GB image, named docker-registry.discovery.wmnet/mwcachedir:0.0.1, containing the cache/ dir of a mediawiki installation to the registry. T264209 [production]
14:01 <akosiaris> push a 6GB image, named docker-registry.discovery.wmnet/mwcachedir:0.0.1, containing the cache/ dir of a mediawiki installation to the registry. T265183 [production]
13:53 <jbond42> enable puppet fleet wide post - convert puppetdb stockpile queue to tmpfs [production]
13:48 <jbond42> disable puppet fleet wide to convert puppetdb stockpile queue to tmpfs [production]
12:46 <vgutierrez> Bump ECDHE-ECDSA-AES128-SHA pageview replacement to 10% - T258405 [production]
11:50 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
11:50 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]