9251-9300 of 10000 results (25ms)
2017-10-04 §
17:37 <elukey> enabled basic ACLs on the Kafka Jumbo cluster - T173493 [production]
12:10 <elukey> added two new mediawiki videoscalers - mw1307/1318 [production]
12:09 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet [production]
12:09 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet [production]
09:45 <elukey> rolling restart of aqs nodes to pick up the new logstash lvs config [production]
06:21 <elukey> restart varnish backend on cp3043 [production]
06:16 <elukey> restart varnish backend on cp3041 [production]
2017-10-03 §
07:05 <elukey> restart varnish backend on cp3031 (503s) [production]
2017-10-02 §
12:56 <elukey> added two new mediawiki jobrunners - mw131[01] [production]
10:06 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1311.eqiad.wmnet [production]
10:06 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1310.eqiad.wmnet [production]
09:06 <elukey> forced remount of /mnt/hdfs on notebook1002 [production]
06:19 <elukey> restart varnish backend on cp3040 [production]
06:10 <elukey> restart varnish backend on cp3033 [production]
2017-10-01 §
13:50 <elukey> restart hhvm on mw1167 (jobrunner) - hhvm stuck, dump-debug in /tmp/hhvm.9624.bt. [production]
2017-09-29 §
13:09 <elukey> depool mw1265 (hhvm 3.18.5) - disk filled up [production]
11:44 <elukey> re-enable job runner daemons on mw130[8,9] [production]
11:05 <elukey> precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation [production]
09:55 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet [production]
2017-09-28 §
10:09 <elukey> incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally) [production]
10:00 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet [production]
09:59 <elukey> added new mediawiki jobrunner - mw1308 [production]
08:00 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet [production]
06:28 <elukey> restart varnish backend on cp3032 [production]
2017-09-27 §
14:45 <elukey> rolling restart of all the Yarn nodemanager daemons on analytics1028-1068 [production]
13:42 <elukey> raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12] [production]
08:34 <elukey> raise traffic weights to 30 for mw13[19-28] incrementally - T165519 [production]
06:15 <elukey> restart varnish backend on cp3043 [production]
2017-09-26 §
13:00 <elukey> add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20 [production]
09:12 <elukey> add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20 [production]
2017-09-25 §
10:14 <elukey> add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally [production]
07:37 <elukey> add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally [production]
2017-09-22 §
13:49 <elukey> mw1321 (new appserver) serving traffic (going to increase its weight up to 20) [production]
09:42 <elukey> mw1319 (new appserver) serving traffic (going to increase its weight up to 20) [production]
2017-09-19 §
09:40 <elukey> powercycle analytics1062 - no ssh, console com2 frozen [production]
2017-09-17 §
09:46 <elukey> restart varnish backend on cp1052 - recurrent mailbox lag [production]
08:11 <elukey> restart varnish backend on cp1053 - recurrent mailbox lag [production]
2017-09-16 §
16:53 <elukey> restart varnish-backend on cp1073 (cache upload) for mailbox lag [production]
2017-09-13 §
13:29 <elukey> update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts [production]
08:55 <elukey> restart varnish backend on cp1072 (upload) - mailbox expiry lag [production]
2017-09-12 §
14:43 <elukey> add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad [production]
2017-09-10 §
13:41 <elukey> restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag [production]
13:13 <elukey> restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473 [production]
2017-09-07 §
08:44 <elukey> restart varnish backend on cp1063 - mailbox expiry lag [production]
07:57 <elukey> force re-mount of /mnt/hdfs on stat1005 [production]
2017-09-06 §
11:24 <elukey> temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493 [production]
2017-09-05 §
16:58 <elukey> ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org [production]
16:45 <elukey> add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload) [production]
2017-09-04 §
12:10 <elukey> rolling restart of zookeeper on conf100[123] for jvm security updates [production]
08:52 <elukey> restart zookeeper on conf2003 for jvm security updates [production]