9101-9150 of 10000 results (26ms)
2018-01-08 §
09:39 <elukey> set sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 to mw1261,mw2251,mw1276 and all videoscalers (Recently rebooted/reimaged) [production]
2018-01-07 §
08:52 <elukey> re-enabled puppet on db110[78] - eventlogging_sync restarted on db1108 (analytics-slave) - T168414 [production]
2018-01-06 §
08:09 <elukey> re-enable eventlogging mysql consumers after database maintenance - T168414 [production]
06:59 <elukey> set sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on mw[1329-1333] (new appservers, was 120) [production]
06:49 <elukey> set sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on mw1335 (new jobrunner, was 120) [production]
2018-01-05 §
13:57 <elukey@tin> Finished deploy [analytics/aqs/deploy@792c95d]: Add pageviews by country endpoint (duration: 01m 12s) [production]
13:56 <elukey@tin> Started deploy [analytics/aqs/deploy@792c95d]: Add pageviews by country endpoint [production]
12:44 <elukey> reboot kafka-jumbo1001 for kernel updates [production]
09:20 <elukey> drain and reboot analytics1030 for kernel updates [production]
08:54 <elukey> ran git checkout modules/role/manifests/puppetmaster/standalone.pp on labs-puppetmaster.wikimedia.org to unblock sync from prod [production]
2018-01-04 §
08:57 <elukey> set sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on mw133[67] (new jobrunners) [production]
2018-01-03 §
17:07 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw2251.*.wmnet [production]
15:38 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mw2251.*.wmnet [production]
11:03 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1336.*.eqiad.wmnet [production]
09:57 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1337.*.eqiad.wmnet [production]
08:59 <elukey> stop eventlogging mysql insertion on eventlog1001 to allow db1107 maintenance - T168414 [production]
2018-01-02 §
16:48 <elukey@puppetmaster1001> conftool action : set/weight=30; selector: name=mw13(29|3[0-3]).*.eqiad.wmnet [production]
15:53 <elukey@puppetmaster1001> conftool action : set/weight=20; selector: name=mw13(29|3[0-3]).*.eqiad.wmnet [production]
15:33 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1335.*.eqiad.wmnet [production]
13:49 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1333.*.eqiad.wmnet [production]
13:48 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1332.*.eqiad.wmnet [production]
13:47 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1331.*.eqiad.wmnet [production]
13:45 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1330.*.eqiad.wmnet [production]
13:42 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1329.*.eqiad.wmnet [production]
13:41 <elukey> enable live traffic for new appservers mw1329->mw1333 (T165519) [production]
08:23 <elukey> restart druid coordinators on druid* to pick up new jvm settings [production]
2017-12-21 §
12:39 <elukey> restart druid historical/broker to apply new jvm settings to druid public workers (druid100[456] - https://gerrit.wikimedia.org/r/399617) [production]
12:07 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1333.*.eqiad.wmnet [production]
10:15 <elukey> rolling restart of eventbus on kafka* for openssl security updates [production]
09:50 <elukey> restart eventbus on kafka2001 for openssl updates [production]
09:30 <elukey> restart zookeeper on conf100[2,3] for jvm updates - T179943 [production]
09:20 <elukey> restart zookeeper on conf1001 for jvm updates - T179943 [production]
08:51 <elukey> run kafka preferred-replica-election after maintenance of kafka1023 (fully bootstrapped now) [production]
08:23 <elukey> repool mw1277 after investigation [production]
2017-12-20 §
17:48 <elukey> new mw jobrunner in production (mw1334) - T165519 [production]
17:43 <elukey> restart zookeeper on conf2003 for jvm updates - T179943 [production]
17:42 <elukey> restart hhvm on mw1316 [production]
17:38 <elukey> restart hhvm on mw1234 [production]
17:32 <elukey> restart zookeeper on conf2002 for jvm updates - T179943 [production]
15:47 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1334.eqiad.wmnet [production]
14:06 <elukey> temporarily shutdown kafka on kafka1023 to move some topic partitions on different disk partition (disk space usage alerts) [production]
13:49 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1276.eqiad.wmnet [production]
13:38 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1276.eqiad.wmnet [production]
2017-12-19 §
16:34 <elukey> manually started eventlogging cleaner on db1107 to purge/sanitize data up to 90 days ago (tmux is running for user eventlogcleaner) - T108850 [production]
13:05 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw133[0-1].eqiad.wmnet [production]
10:53 <elukey> reboot conf2001 for kernel updates - T179943 [production]
10:47 <elukey> restart zookeeper on conf2001 for jvm updates - T179943 [production]
2017-12-18 §
17:51 <elukey> run kafka preferred-replica-election on the analytics cluster to allow kafka1023 (new node) to become a partition leader [production]
16:15 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mw133[0-7].eqiad.wmnet [production]
15:28 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1329.eqiad.wmnet [production]