4351-4400 of 10000 results (54ms)
2018-06-19 §
14:11 <herron> increased client_max_body_size on puppetdb nginx frontends from 30m to 60m https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/ [production]
14:06 <herron> temporarily disabling puppet agents for deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/ [production]
12:41 <_joe_> hard reboot of ms-be1019 - unable to ssh, console showing i/o errors only [production]
04:56 <ebernhardson> unban elastic1035 [production]
04:45 <ebernhardson> ban elastic1035 from cluster to allow it to recover [production]
02:36 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.999) (duration: 06m 53s) [production]
02:19 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 39s) [production]
2018-06-18 §
19:49 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@bcb2904]: GUI update (duration: 18m 52s) [production]
19:30 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@bcb2904]: GUI update [production]
19:30 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@37f6f32]: GUI update (duration: 00m 56s) [production]
19:29 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@37f6f32]: GUI update [production]
18:41 <Trey314159> reindexing Serbian wikis on elastic@eqiad (T196404) [production]
16:39 <urandom> DROP unused Cassandra keyspaces - T197080 [production]
10:52 <_joe_> removing wtp1043 from all pybal configuration until the disk is replaced T196886 [production]
10:04 <_joe_> initialize namespace "ci" on the kubernetes staging cluster T196654 [production]
09:56 <joal@deploy1001> Finished deploy [analytics/refinery@e9dbe79]: Regular weekly deploy (duration: 06m 54s) [production]
09:49 <joal@deploy1001> Started deploy [analytics/refinery@e9dbe79]: Regular weekly deploy [production]
03:04 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.999) (duration: 14m 30s) [production]
02:33 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 12s) [production]
02:27 <Trey314159> reindexing Serbian wikis on elastic@codfw (T196404) [production]
2018-06-17 §
22:24 <no_justification> gerrit: started back up, nvm [production]
22:23 <no_justification> gerrit: stopping services for a few minutes [production]
13:05 <Trey314159> reindexing Serbo-Croatian wikis on elastic@eqiad (T196658) [production]
03:46 <Trey314159> reindexing Serbo-Croatian wikis on elastic@codfw (T196658) [production]
2018-06-16 §
20:42 <mdholloway> disabled puppet on maps-test2004 for testing new map styles setup [production]
19:24 <Trey314159> reindexing Croatian wikis on elastic@eqiad (T196658) [production]
13:19 <Trey314159> reindexing Croatian wikis on elastic@codfw (T196658) [production]
00:15 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Ica4cb6644 (duration: 00m 59s) [production]
2018-06-15 §
18:48 <Trey314159> reindexing Bosnian wikis on elastic@eqiad (T196658) [production]
18:10 <Trey314159> reindexing Bosnian wikis on elastic@codfw (T196658) [production]
17:15 <krinkle@deploy1001> Synchronized wmf-config/mc.php: I619a2ff5db611 (duration: 00m 58s) [production]
15:49 <mutante> install2002 - disabling puppet temp, live hackking DHCP config for debugging backup2001 install issue [production]
15:37 <mutante> switching noc.wikimedia.org site from terbium to mwamiant1001 backend, running puppet on all cache::misc cp servers (T192092) [production]
15:19 <ottomata> bouncing kafka broker on kafka-jumbo1001 to test https://gerrit.wikimedia.org/r/#/c/440520/ [production]
15:19 <gehel> rolling restart of elasticsearch eqiad for plugin upgrade completed - T194245 [production]
14:49 <elukey> restart varnishkafka-eventlogging on cp4028, errors logged [production]
14:43 <elukey> restart varnishkafka-eventlogging on cp5012 as attempt to clear out the errors (not needed but logging it anyway) [production]
14:12 <papaul> OS install on backup2001 [production]
13:41 <mepps> updated process control to 3b9f51e0fd [production]
12:56 <legoktm@deploy1001> Synchronized wmf-config/mc.php: Make sure that mcrouter BagOStuff goes through ObjectCache::newFromParams() - T197450 (duration: 00m 57s) [production]
12:38 <jynus> killing refresh counts on commonswiki [production]
12:30 <jynus> reenabling db2040 consistency after slaves caught up [production]
12:13 <papaul> OS install on bast2002 [production]
10:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 58s) [production]
09:48 <godog> delete cpjobqueue metrics older than 10d - T196067 [production]
09:45 <jynus> reenabling db2048 consistency after slaves caught up [production]
09:43 <jynus> reducing temp. db2040 consistency to speed up slave lag catch up [production]
09:20 <godog> fully remove ms-be1036 from swift due to hw failure - T196873 [production]
09:03 <bawolff> deploy patch T197279 [production]
08:05 <gehel> rolling restart of elasticsearch eqiad for plugin upgrade - T194245 [production]