4001-4050 of 10000 results (60ms)
2018-06-20 §
08:54 <_joe_> killall nsca on tegmen [production]
08:51 <_joe_> restarting nsca daemon on tegmen (gone wild, hundreds of subprocesses [production]
08:47 <_joe_> removing user.log.1 and messages.log.1 on tegmen to save some space [production]
07:40 <_joe_> powercycled mw1272, down since yesterday [production]
07:29 <hashar> deploy1001: rebased php-1.32.0-wmf.8/extensions/Translate to catch up with a non production merged change ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/441127 ). [production]
02:20 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 17s) [production]
2018-06-19 §
16:10 <mobrovac@deploy1001> Finished deploy [proton/deploy@43af7d9]: (no justification provided) (duration: 00m 27s) [production]
16:09 <mobrovac@deploy1001> Started deploy [proton/deploy@43af7d9]: (no justification provided) [production]
14:13 <herron> re-enabling puppet agents [production]
14:11 <herron> increased client_max_body_size on puppetdb nginx frontends from 30m to 60m https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/ [production]
14:06 <herron> temporarily disabling puppet agents for deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/ [production]
12:41 <_joe_> hard reboot of ms-be1019 - unable to ssh, console showing i/o errors only [production]
04:56 <ebernhardson> unban elastic1035 [production]
04:45 <ebernhardson> ban elastic1035 from cluster to allow it to recover [production]
02:36 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.999) (duration: 06m 53s) [production]
02:19 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 39s) [production]
2018-06-18 §
19:49 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@bcb2904]: GUI update (duration: 18m 52s) [production]
19:30 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@bcb2904]: GUI update [production]
19:30 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@37f6f32]: GUI update (duration: 00m 56s) [production]
19:29 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@37f6f32]: GUI update [production]
18:41 <Trey314159> reindexing Serbian wikis on elastic@eqiad (T196404) [production]
16:39 <urandom> DROP unused Cassandra keyspaces - T197080 [production]
10:52 <_joe_> removing wtp1043 from all pybal configuration until the disk is replaced T196886 [production]
10:04 <_joe_> initialize namespace "ci" on the kubernetes staging cluster T196654 [production]
09:56 <joal@deploy1001> Finished deploy [analytics/refinery@e9dbe79]: Regular weekly deploy (duration: 06m 54s) [production]
09:49 <joal@deploy1001> Started deploy [analytics/refinery@e9dbe79]: Regular weekly deploy [production]
03:04 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.999) (duration: 14m 30s) [production]
02:33 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 12s) [production]
02:27 <Trey314159> reindexing Serbian wikis on elastic@codfw (T196404) [production]
2018-06-17 §
22:24 <no_justification> gerrit: started back up, nvm [production]
22:23 <no_justification> gerrit: stopping services for a few minutes [production]
13:05 <Trey314159> reindexing Serbo-Croatian wikis on elastic@eqiad (T196658) [production]
03:46 <Trey314159> reindexing Serbo-Croatian wikis on elastic@codfw (T196658) [production]
2018-06-16 §
20:42 <mdholloway> disabled puppet on maps-test2004 for testing new map styles setup [production]
19:24 <Trey314159> reindexing Croatian wikis on elastic@eqiad (T196658) [production]
13:19 <Trey314159> reindexing Croatian wikis on elastic@codfw (T196658) [production]
00:15 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Ica4cb6644 (duration: 00m 59s) [production]
2018-06-15 §
18:48 <Trey314159> reindexing Bosnian wikis on elastic@eqiad (T196658) [production]
18:10 <Trey314159> reindexing Bosnian wikis on elastic@codfw (T196658) [production]
17:15 <krinkle@deploy1001> Synchronized wmf-config/mc.php: I619a2ff5db611 (duration: 00m 58s) [production]
15:49 <mutante> install2002 - disabling puppet temp, live hackking DHCP config for debugging backup2001 install issue [production]
15:37 <mutante> switching noc.wikimedia.org site from terbium to mwamiant1001 backend, running puppet on all cache::misc cp servers (T192092) [production]
15:19 <ottomata> bouncing kafka broker on kafka-jumbo1001 to test https://gerrit.wikimedia.org/r/#/c/440520/ [production]
15:19 <gehel> rolling restart of elasticsearch eqiad for plugin upgrade completed - T194245 [production]
14:49 <elukey> restart varnishkafka-eventlogging on cp4028, errors logged [production]
14:43 <elukey> restart varnishkafka-eventlogging on cp5012 as attempt to clear out the errors (not needed but logging it anyway) [production]
14:12 <papaul> OS install on backup2001 [production]
13:41 <mepps> updated process control to 3b9f51e0fd [production]
12:56 <legoktm@deploy1001> Synchronized wmf-config/mc.php: Make sure that mcrouter BagOStuff goes through ObjectCache::newFromParams() - T197450 (duration: 00m 57s) [production]
12:38 <jynus> killing refresh counts on commonswiki [production]