3501-3550 of 10000 results (66ms)
2019-02-22 §
09:33 <gehel@cumin2001> END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) [production]
09:33 <moritzm> installing tor security update on torrelay1001 [production]
09:33 <gehel@cumin2001> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
09:32 <_joe_> set pooled=inactive on mw1272, T211668 [production]
09:26 <gehel@cumin2001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
09:22 <gilles@deploy1001> Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 16s) [production]
09:22 <gilles@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
09:22 <moritzm> updated tor packages to 0.3.5.8-1~d90.stretch+1 [production]
09:18 <gehel@cumin2001> END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) [production]
09:16 <gilles@deploy1001> Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 14s) [production]
09:16 <gilles@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
09:16 <gehel@cumin2001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
09:16 <gehel> starting rolling upgrade on elasticsearch / cirrus / eqiad - T215931 [production]
08:52 <godog> force ftpsync run on sodium after debian mirror update [production]
08:19 <moritzm> installing uriparser security updates [production]
08:18 <godog> temporarily stop prometheus global on prometheus2004 to take a snapshot [production]
07:47 <moritzm> installing krb5 updates for jessie [production]
07:46 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool es1013 after MySQL upgrade (duration: 00m 46s) [production]
07:28 <elukey> manually delete WANCache:v:metawiki:translate-groups from memcache on mc1022 to test fix for T203786 [production]
07:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Give more traffic to es1013 after MySQL upgrade (duration: 00m 45s) [production]
07:15 <_joe_> deactivating mw1272, memory problems [production]
07:03 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool es1013 after MySQL upgrade (duration: 00m 45s) [production]
06:51 <marostegui> Power cycle mw1272 as it crashed - T211668 [production]
06:49 <marostegui> Stop MySQL on es1013 to upgrade MySQL [production]
06:48 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool es1013 for MySQL upgrade (duration: 02m 50s) [production]
06:30 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1087 after MySQL upgrade (duration: 02m 51s) [production]
06:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1087 for MySQL upgrade (duration: 02m 53s) [production]
06:15 <marostegui> Stop MySQL on db1087 for kernel and mysql upgrade [production]
03:26 <XioNoX> delete old gr-1/0/0 from cr1-eqsin - T213121 [production]
01:58 <XioNoX> power-down cp5007 - T216716 [production]
01:40 <XioNoX> power-down cp5006 - T216717 [production]
00:57 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Noop sync of labs settings (duration: 00m 44s) [production]
00:46 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T215931 [cirrus] Switch production search traffic to codfw (2/2) (duration: 00m 46s) [production]
00:45 <ebernhardson@deploy1001> sync-file aborted: T215931 [cirrus] Switch production search traffic to codfw (2/2) (duration: 00m 05s) [production]
00:39 <ebernhardson@deploy1001> Synchronized wmf-config/Wikibase.php: Deploy WikibaseCirrusSearch: Part III, Wikibase.php (duration: 00m 45s) [production]
00:27 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Deploy WikibaseCirrusSearch: Part II, InitialiseSettings.php (duration: 00m 46s) [production]
00:23 <ebernhardson@deploy1001> Synchronized wmf-config/extension-list: Deploy WikibaseCirrusSearch: Part I, extensionlist (duration: 00m 46s) [production]
00:21 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T215931 [cirrus] Switch production search traffic to codfw (1/2) (duration: 00m 45s) [production]
00:18 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T215931 [cirrus] Switch production search traffic to codfw (1/2) (duration: 00m 46s) [production]
00:17 <ebernhardson@deploy1001> sync-file aborted: T215931 (duration: 00m 00s) [production]
2019-02-21 §
22:25 <tzatziki> change pw for NazarSusP [production]
22:17 <volans> forcing a puppet run on A:ganeti [production]
20:35 <gehel@cumin2001> END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0) [production]
20:18 <thcipriani@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.18 [production]
20:06 <ladsgroup@deploy1001> Finished deploy [ores/deploy@5d937b1]: Drop accepting pickle altogether (T206333) (duration: 13m 17s) [production]
19:58 <bblack> eqsin: repooling user traffic [production]
19:52 <ladsgroup@deploy1001> Started deploy [ores/deploy@5d937b1]: Drop accepting pickle altogether (T206333) [production]
19:35 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:491506|Drop obsolete Wikibase configs (T213713)]], Part II (duration: 00m 53s) [production]
19:33 <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:491506|Drop obsolete Wikibase configs (T213713)]], Part I (duration: 00m 52s) [production]
19:32 <gehel@cumin2001> END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) [production]