2851-2900 of 10000 results (31ms)
2015-08-28 §
01:58 <TimStarling> on ruthenium, reduced parsoid-rt-client concurrency from 16 to 8 since it was OOM and oom-killer was killing random things [production]
01:37 <TimStarling> on ruthenium restarted parsoid-rt-client and parsoid-vd-client [production]
00:24 <mutante> powercycled mw2027 [production]
00:19 <krenair@tin> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234450/ (duration: 01m 14s) [production]
00:05 <krenair@tin> Synchronized wmf-config/mobile.php: live hack to make previous commit work (duration: 01m 14s) [production]
00:05 <Krenair> Another codfw host broke: mw2027 [production]
00:01 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234330/ (duration: 00m 13s) [production]
2015-08-27 §
23:58 <krenair@tin> Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/MobileFormatter.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 12s) [production]
23:57 <krenair@tin> Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/config/Experimental.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 14s) [production]
23:55 <krenair@tin> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233439/ (duration: 00m 12s) [production]
23:29 <krenair@tin> Synchronized php-1.26wmf20/extensions/Gadgets/extension.json: touch (duration: 00m 13s) [production]
23:24 <krenair@tin> Synchronized php-1.26wmf20/includes/DefaultSettings.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s) [production]
23:24 <krenair@tin> Synchronized php-1.26wmf20/includes/registration/ExtensionProcessor.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s) [production]
23:23 <krenair@tin> Synchronized php-1.26wmf20/includes/MWNamespace.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 13s) [production]
23:15 <krenair@tin> Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/234009/ (duration: 00m 13s) [production]
23:04 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233100/ (duration: 00m 12s) [production]
20:11 <chasemp> ferm setup on elasticsearch10(1[8-9|2[0-3]) [production]
20:06 <twentyafterfour@tin> rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf20 [production]
19:57 <twentyafterfour@tin> Synchronized php-1.26wmf20/includes/media/XMP.php: deploy fix for T89532 on 1.26wmf20 (duration: 00m 13s) [production]
18:16 <chasemp> setting up ferm on elastic1027-31 [production]
17:47 <krenair@tin> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 13s) [production]
17:43 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/2 (duration: 00m 13s) [production]
17:37 <urandom> ack'd Cassandra process alert on restbase1001; temporary command args have pushed the class name beyond the limit [production]
17:34 <krenair@tin> Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s) [production]
17:24 <krenair@tin> Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 12s) [production]
17:08 <urandom> bouncing Cassandra on restbase1001 to apply temporary GC settings [production]
16:51 <moritzm> ferm rules on logstash100[1-3] have been amended to allow grafana from reading dashboard configs [production]
16:39 <bd808> new ferm rules on logstash100[1-3] are blocking grafana from reading dashboard configs. [production]
16:22 <moritzm> ferm enabled on logstash1003 [production]
16:18 <moritzm> ferm enabled on logstash1002 [production]
16:16 <bd808> ferm enabled on logstash1001 [production]
16:06 <bd808> logstash1001 back up after system reboot; we applied a default drop rule without applying the other iptables changes; will try again [production]
15:58 <chasemp> rebooting logstash1001.mgmt.eqiad.wmnet for moritz as it is having issues [production]
15:47 <bblack> killed hung ubuntu mirror rsync commands on carbon, from Jul 10 [production]
15:45 <bd808> logstash1001 not responding over ssh following ferm rules application; moritzm investigating [production]
15:30 <bd808> Disabled puppet on logstash100[1-3] prior to trying to enable ferm [production]
15:11 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable newarticle campaign in itwiki [[gerrit:234223]] (duration: 01m 52s) [production]
14:52 <bblack> re-imaging lvs200[123] [production]
14:47 <godog> reenable puppet on ms-be1* [production]
14:22 <godog> disable puppet on ms-fe1 / ms-be1 in prepration for puppet work [production]
14:15 <godog> reenable puppet on ms-fe2* [production]
13:47 <bblack> re-imaging lvs2004 + lvs2005 [production]
13:29 <ottomata> doing rolling restart of kafka brokers to apply auto_create_topics change [production]
13:21 <godog> enable puppet on ms-be2* [production]
13:21 <ottomata> stopping kafka on analytics1021, it is no longer a kafka broker. [production]
13:09 <godog> disable puppet on ms-be2* in preparation for firewall changes [production]
13:09 <jynus> cloning es1008 into es1014 [production]
13:04 <ottomata> running leader election now that all topics and partitions are rebalanced across new kafka nodes [production]
12:46 <bblack> re-imaging lvs2006 [production]
12:45 <andrewbogott> re-imaging labnet1001 (I hope) [production]