2201-2250 of 3409 results (14ms)
2018-03-26 §
14:26 <ottomata> restarting jumbo -> eqiad mirror makers with prometheus instead of jmx [analytics]
13:28 <ottomata> restarting kafka mirror maker main -> jumbo using new consumer [analytics]
13:09 <fdans> stopped 2 mysql consumers as precaution for T174386 [analytics]
2018-03-24 §
08:13 <joal> kill failing query swamping the cluster (application_1520532368078_47226) [analytics]
2018-03-23 §
16:44 <elukey> invalidated 2018-03-12/13 for iOS data in piwik to force a re-run of the archiver [analytics]
2018-03-20 §
10:10 <elukey> removed old mysql/ssh/ganglia analytics vlan firewall rules (https://phabricator.wikimedia.org/T189408#4055749) [analytics]
2018-03-19 §
09:38 <elukey> restart hadoop daemons on analytics1070 for openjdk upgrades (canary) [analytics]
2018-03-16 §
20:23 <ottomata> bouncing main -> jumbo mirror makers to apply change-prop topic blacklist [analytics]
14:44 <ottomata> restarting eventlogging mysql eventbus consumer to consume from analytics instead of jumbo [analytics]
14:38 <elukey> temporary point pivot to druid1002 as prep step for druid1001's reboot [analytics]
14:37 <elukey> disable druid1001's middlemanager as prep step for reboot [analytics]
14:24 <elukey> changed superset druid private config from druid1002 to druid1003 [analytics]
13:43 <elukey> disable druid1002's middle manager via API as prep step for reboot [analytics]
09:57 <elukey> restart eventlogging-consumer@mysql-m4/eventbus on eventlog1002 to force the DNS resolution of m4-master (changed from dbproxy1009 -> dbproxy1004) [analytics]
2018-03-15 §
22:13 <ottomata> bounced jumbo mirror makers [analytics]
19:10 <ottomata> bouncing main -> jumbo mirror maker [analytics]
14:50 <joal> Restart clickstream-coord to pick new config including fawiki [analytics]
14:29 <elukey> disabled druid1003's middlemanager as prep step for reboot [analytics]
14:07 <ottomata> bouncing kafka jumob -> eqiad mirrormaker [analytics]
2018-03-14 §
15:27 <ottomata> bouncing main -> jumbo mirror maker instances [analytics]
14:45 <ottomata> beginning migration of eventlogging analtyics from Kafka analytics to Kafka jumbo: T183297 [analytics]
2018-03-13 §
20:47 <ottomata> restarting eventlogging processors to pick up VirtualPageView blacklist from eventlogging-valid-mixed topic [analytics]
15:13 <ottomata> bounce main -> analytics mirror maker instances [analytics]
15:07 <ottomata> bouncing MirrorMaker on kafka1020 (main -> jumbo) to re-apply acks=all [analytics]
14:55 <ottomata> bouncing MirrorMaker on kafka1022 to re-apply acks=all (main -> jumbo) [analytics]
14:32 <ottomata> bouncing MirrorMaker on kafka1023 (main -> jumbo) to re-apply acks=all [analytics]
14:22 <ottomata> bouncing mirrormaker for main -> analytics on kafka101[234] to apply roundrobin [analytics]
2018-03-12 §
19:39 <ottomata> deployed new Refine jobs (eventlogging, eventbus, etc.) with deduplication and geocoding and casting [analytics]
18:17 <ottomata> bouncing kafka mm eqiad -> jumbo witih acks=1 [analytics]
18:10 <ottomata> bouncing kafka mirrormaker for main-eqiad -> jumbo with buffer.memory=128M [analytics]
17:34 <joal> Restart mediawiki-history-reduced oozie job to add a dependency [analytics]
16:55 <joal> Restart mobile_apps_session_metrics [analytics]
16:52 <joal> Deploying refinery on HDFS for mobile_apps patch [analytics]
16:26 <joal> Deploying refinery again to provide patch for mobile_apps_session_metric job [analytics]
15:09 <joal> Deploy refinery onto hdfs [analytics]
15:07 <joal> Deploy refinery from scap [analytics]
14:32 <elukey> restart druid-broker on druid1004 - no /var/log/druid/broker.log after 2018-03-10T22:38:52 (java.io.IOException: Too many open files_ [analytics]
08:50 <elukey> fixed evenglog1002's ipv6 (https://gerrit.wikimedia.org/r/#/c/418714/) [analytics]
2018-03-10 §
09:07 <joal> Rerun clickstream-wf-2018-2 [analytics]
00:32 <milimetric> finished sqooping pagelinks for missing dbs, hdfs -put a SUCCESS flag in the 2018-02 snapshot, jobs should run unless Hue is still lying to itself [analytics]
2018-03-09 §
17:29 <joal> Rerun mediawiki-history-reduced job after having manually repaired wmf_raw.mediawiki_project_namespace_map [analytics]
2018-03-08 §
18:05 <ottomata> bouncing ResourceManagers [analytics]
08:54 <elukey> re-enable camus after reboots [analytics]
07:15 <elukey> disable Camus on an1003 to allow the cluster to drain - prep step for an100[123] reboot [analytics]
2018-03-07 §
07:15 <elukey> manually re-run wikidata-articleplaceholder_metrics-wf-2018-3-6 [analytics]
2018-03-06 §
20:44 <ottomata> reverted change to point mediawiki monolog kafka producers at kafka jumbo-eqiad until deployment train is done T188136 [analytics]
20:35 <ottomata> pointing mediawiki monolog kafka producers at kafka jumbo-eqiad cluster: T188136 [analytics]
19:06 <elukey> cleaned up id=0 rows on db1108 (log database) for T188991 [analytics]
10:19 <elukey> restart webrequest-load-wf-upload-2018-3-6-7 (failed due to reboots) [analytics]
10:08 <elukey> re-starting mysql consumers on eventlog1001 [analytics]