4051-4100 of 4917 results (25ms)
2017-08-29 §
19:36 <ottomata> restarting all kafka brokers and mirror maker processes to apply https://gerrit.wikimedia.org/r/#/c/374610/ [analytics]
12:46 <elukey> suspend oozie jobs from Hue to allow a easier restart of oozie/hive daemons [analytics]
2017-08-28 §
13:57 <elukey> restart kafka* on kafka1012 for openjdk security updates (canary) [analytics]
10:34 <elukey> restart yarn and hdfs on analytics1030 for jvm updates (canary) [analytics]
2017-08-23 §
19:50 <joal> restart oozie webrequest-load bundle after bug correction [analytics]
19:46 <joal> Deploy refinery onto hdfs [analytics]
19:41 <joal> Deploying refinery from tin [analytics]
19:36 <joal> Deployed werbrequest-source using jenkins for bug correction [analytics]
19:26 <joal> Alter wmf.webrequest and wmf.wdqs_extract tables to correct bug [analytics]
19:25 <joal> Kill oozie webrequest-load bundle for redeploy after bug correction [analytics]
11:04 <joal> Update wmf.wdqs_extract table for normalized_host update [analytics]
10:12 <joal> Restart oozie webrequest-load bundle after deploy and updates [analytics]
10:09 <joal> Alter webrequest table before restarting oozie load bundle [analytics]
10:06 <joal> Deploying refinery onto hdfs [analytics]
09:59 <joal> Deploying refinery [analytics]
09:59 <joal> Kill oozie webrequest-load bundle for restart after deploy [analytics]
08:25 <joal> Deploying refinery-source v0.0.50 using jenkins [analytics]
2017-08-22 §
19:52 <joal> Drop / recreate wmf.mediawiki_history table for naming correction [analytics]
13:57 <ottomata> sudo -u hdfs hdfs dfs -rm /tmp/druid-indexing/classpath/guava.jar (guava 11.0.2 is conflicting with guava 16.0.1. from druid-hdfs-storage-cdh extension). Not sure how guava 11.0.2 got there, but let's see if it doesn't come back [analytics]
08:27 <joal> Rerun druid loading jobs after night failures [analytics]
2017-08-21 §
13:46 <ottomata> adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on db1047: T170990 [analytics]
13:26 <ottomata> adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on dbstore1002: T170990 [analytics]
2017-08-14 §
16:40 <elukey> analytics1034 back in service after swapping the eth cable - T172633 [analytics]
2017-08-10 §
20:06 <milimetric> stopped Wikimetrics web and queue on wikimetrics-01.eqiad.wmflabs because the queue ran into errors connecting to the database (max 10 connections limit reached) [analytics]
08:59 <elukey> updated librdkafka1 to 0.9.4.1 on eventlog1001 [analytics]
2017-08-08 §
18:39 <elukey> restart projectview-hourly-wf-2017-8-8-14, pageview-druid-hourly-wf-2017-8-8-14, pageview-hourly-wf-2017-8-8-14 via Hue (analytics1055 disk failure) [analytics]
14:20 <elukey> restart varnishkafka statsv/eventlogging instances to pick up https://gerrit.wikimedia.org/r/#/c/370637/ (kafka protocol explicitly set to 0.9.0.1) [analytics]
2017-08-06 §
11:02 <elukey> stop yarn on analytics1034 to reload the tg3 driver - T172633 [analytics]
2017-08-03 §
16:15 <ottomata> druid cluster restarted with 0.9.2 mysql-metadata-storage extension, un-suspending oozie druid jobs [analytics]
14:11 <ottomata> pausing oozie druid jobs and doing a cluster upgrade/restart again to make sure updated version of mysql-metadata-storage jar is properly loaded [analytics]
09:56 <elukey> set piwik in maintenance mode to allow mysql updates [analytics]
08:08 <elukey> restarted Druid jobs failed over night (drud_loader.py error) and due to Hive metastore restart [analytics]
08:03 <elukey> restart hive-metastore to pick up new JVM Xms settings [analytics]
2017-08-02 §
14:34 <ottomata> beginning druid upgrade to 0.92 (take 2 :) ) [analytics]
14:23 <elukey> restart hive-server to pick up JVM Xms4g change [analytics]
14:22 <ottomata> suspending druid oozie jobs [analytics]
2017-08-01 §
18:57 <madhuvishy> Bumped instance quota to 24 instances (nova quota-update analytics --instances 24) [analytics]
17:24 <ottomata> beginning druid upgrade to 0.9.2 http://druid.io/docs/0.9.2/operations/rolling-updates.html [analytics]
17:10 <ottomata> pausing all druid oozie coordinators [analytics]
12:49 <elukey> restart hive daemons on analytics1003 to pick up new jvm settings (bigger Xmx, JMX ports) [analytics]
10:05 <elukey> suspended again webrequest-load-bundle as prep step to restart the hive daemons [analytics]
07:58 <elukey> suspended webrequest-load-bundle as prep step to restart the hive daemons [analytics]
07:03 <elukey> restarted mobile_apps-session_metrics-coord-global-30days failed job via Hue [analytics]
2017-07-31 §
13:45 <elukey> suspended webrequest-load-bundle as prep step to restart hive metastore/server [analytics]
10:34 <elukey> restart hive-server on an1003 - beeline not connecting, thrift errors [analytics]
2017-07-28 §
07:55 <elukey> update nodejs to 6.11 on aqs1004 (testing prod node after beta qa) [analytics]
07:54 <elukey> re-run webrequest-load-wf-upload-2017-7-28-6 from Hue (was playing with eth0 issues on an1034) [analytics]
02:08 <ottomata> stat1002: disabled puppet, umounted /tmp, /home and /a, poweroff [analytics]
2017-07-26 §
21:01 <mforns> Deployed refinery using scap, then deployed onto hdfs [analytics]
18:57 <mforns> Deployed refinery-source using jenkins [analytics]