801-850 of 2039 results (20ms)
2018-04-03 §
19:32 <ottomata> bouncing main -> jumbo MirrorMaker unsetting http://session.timeout.ms/, this has a restiction on the broker in 0.9 :( [analytics]
19:22 <ottomata> bouncing main -> jumbo MirrorMaker setting session.timeout.ms = 125000 [analytics]
18:46 <ottomata> restart main -> jumbo MirrorMaker with request.timeout.ms = 2 minutes [analytics]
15:26 <elukey> manually run hdfs balancer on an1003 (tmux session) [analytics]
15:25 <elukey> killed a jvm belonging to hdfs-balancer stuck from march 9th [analytics]
13:48 <ottomata> re-enable job queue topic mirroring from main -> eqiad [analytics]
2018-04-02 §
22:28 <ottomata> bounce mirror maker to pick up client_id config changes [analytics]
20:55 <ottomata> deployed multi-instance mirrormaker for main -> jumbo. 4 per host == 12 total processes [analytics]
11:25 <joal> Repair cu_changes hive table afer succesfull sqoop import and add _PARTITIONED file for oozie jobs to launch [analytics]
08:33 <joal> rerun wikidata-specialentitydata_metrics-wf-2018-4-1 [analytics]
2018-03-30 §
13:48 <elukey> restart overlord+middlemanager on druid100[23] to avoid consistency issues [analytics]
13:41 <elukey> restart overlord+middlemanager on druid1001 after failures in real time indexing (overlord leader) [analytics]
09:44 <elukey> re-enable camus [analytics]
08:26 <elukey> stopped camus to drain the cluster - prep for easy restart of analytics1003's jvm daemons [analytics]
2018-03-29 §
20:55 <milimetric> accidentally killed mediawiki-geowiki-monthly-coord, and then restarted it [analytics]
20:12 <ottomata> blacklisted mediawiki.job topics from main -> jumbo MirrorMaker again, don't want to page over the weekend while this still is not stable. T189464 [analytics]
07:30 <joal> Manually reparing hive mediawiki_private_cu_changes table after manual sqooping of 2018-01 data, and add _PARTITIONNED file to the folder [analytics]
2018-03-28 §
19:39 <ottomata> bouncing main -> jumbo mirrormaker to apply increase in consumer num.streams [analytics]
19:21 <milimetric> synced refinery to hdfs (only python changes but just so we have latest) [analytics]
19:20 <joal> Start Geowiki jobs (monthly and druid) starting 2018-01 [analytics]
18:36 <joal> Making hdfs://analytics-hadoop/wmf/data/wmf/mediawiki_private accessible only by analytics-privatedata-users group (and hdfs obviously) [analytics]
18:02 <joal> Kill-Restart mobile_apps-session_metrics (bundle killed, coord started) [analytics]
18:00 <joal> Kill-Restart mediawiki-history-reduced-coord after deploy [analytics]
17:44 <joal> Deploying refinery onto hadoop [analytics]
17:29 <joal> Deploy refinery using scap [analytics]
16:32 <ottomata> bouncing main -> jumbo mirror makers to increase heap size to 2G [analytics]
14:16 <ottomata> re-enabling replication of mediawiki job topics from main -> jumbo [analytics]
2018-03-27 §
14:03 <elukey> consolidate all the zookeeper definition in one 'main-eqiad' one in Horizon -> Project-Analytics [analytics]
11:16 <elukey> kill banner impression job to force a respawn (still using an old jvm) [analytics]
2018-03-26 §
15:12 <elukey> restart eventlogging mysql consumers after maintenance [analytics]
14:26 <ottomata> restarting jumbo -> eqiad mirror makers with prometheus instead of jmx [analytics]
13:28 <ottomata> restarting kafka mirror maker main -> jumbo using new consumer [analytics]
13:09 <fdans> stopped 2 mysql consumers as precaution for T174386 [analytics]
2018-03-24 §
08:13 <joal> kill failing query swamping the cluster (application_1520532368078_47226) [analytics]
2018-03-23 §
16:44 <elukey> invalidated 2018-03-12/13 for iOS data in piwik to force a re-run of the archiver [analytics]
2018-03-20 §
10:10 <elukey> removed old mysql/ssh/ganglia analytics vlan firewall rules (https://phabricator.wikimedia.org/T189408#4055749) [analytics]
2018-03-19 §
09:38 <elukey> restart hadoop daemons on analytics1070 for openjdk upgrades (canary) [analytics]
2018-03-16 §
20:23 <ottomata> bouncing main -> jumbo mirror makers to apply change-prop topic blacklist [analytics]
14:44 <ottomata> restarting eventlogging mysql eventbus consumer to consume from analytics instead of jumbo [analytics]
14:38 <elukey> temporary point pivot to druid1002 as prep step for druid1001's reboot [analytics]
14:37 <elukey> disable druid1001's middlemanager as prep step for reboot [analytics]
14:24 <elukey> changed superset druid private config from druid1002 to druid1003 [analytics]
13:43 <elukey> disable druid1002's middle manager via API as prep step for reboot [analytics]
09:57 <elukey> restart eventlogging-consumer@mysql-m4/eventbus on eventlog1002 to force the DNS resolution of m4-master (changed from dbproxy1009 -> dbproxy1004) [analytics]
2018-03-15 §
22:13 <ottomata> bounced jumbo mirror makers [analytics]
19:10 <ottomata> bouncing main -> jumbo mirror maker [analytics]
14:50 <joal> Restart clickstream-coord to pick new config including fawiki [analytics]
14:29 <elukey> disabled druid1003's middlemanager as prep step for reboot [analytics]
14:07 <ottomata> bouncing kafka jumob -> eqiad mirrormaker [analytics]
2018-03-14 §
15:27 <ottomata> bouncing main -> jumbo mirror maker instances [analytics]