701-750 of 2017 results (18ms)
2018-05-02 §
16:17 <joal> Restart oozie mediawiki-history-denormalize job after deploy [analytics]
16:14 <ottomata> bounced eventlogging-consumer@mysql-m4-master-00 after kafka jumbo 1.1.0 upgrade [analytics]
16:05 <joal> Restart oozie webrequest bundle after deploy [analytics]
15:20 <joal> Deploying refinery to hadoop [analytics]
14:45 <joal> Deploying refinery using Scap [analytics]
14:16 <joal> Refinery-source version 0.0.63 finally released to Archiva! [analytics]
13:49 <ottomata> beginning upgrade of kafka-jumbo brokers from 1.0.0 -> 1.1.0 : T193495 [analytics]
13:20 <elukey> restart druid broker on druid100[1-3] to enable druid.sql.enable: true [analytics]
2018-05-01 §
15:33 <elukey> restart historical on druid1003 - exceptions in the logs [analytics]
15:22 <elukey> restart druid-historical on druid1002 - Caused by: java.lang.IllegalArgumentException: Could not resolve type id 'hdfs' into a subtype of [analytics]
11:44 <joal> False positive only in webrequest-load-check_sequence_statistics-wf-upload-2018-5-1-6 [analytics]
07:14 <joal> Rerun webrequest-druid-daily-wf-2018-4-30 [analytics]
06:24 <elukey> roll restart of all middlemanagers on druid100[123] - realtime tasks piled up from hours [analytics]
2018-04-30 §
23:04 <ottomata> blacklisting change-prop and job queue topics from main-eqiad -> analytics (eqiad) [analytics]
22:55 <ottomata> bouncing kafka main-eqiad -> eqiad (analytics) mirror maker [analytics]
19:34 <joal> Retry releasing refinery-source to archiva [analytics]
18:43 <joal> Releasing refinery-source [analytics]
15:53 <joal> Resume webrequest-druid-hourly-coord and pageview-druid-hourly-coord [analytics]
14:23 <joal> Suspend webrequest-druid-hourly-coord and pageview-druid-hourly-coord before druid upgrade [analytics]
14:23 <elukey> disabled cron/check on analytics1003 to respawn banner impressions if needed [analytics]
14:21 <joal> Kill BannerImpressionStream job before upgrading druid [analytics]
2018-04-25 §
14:39 <elukey> re-enable camus after maintenance [analytics]
14:37 <elukey> restart hive-server2 on analytics1003 to pick up settings in https://gerrit.wikimedia.org/r/428919 [analytics]
13:40 <elukey> stop camus on an1003 as prep step to gracefully restart hive server [analytics]
12:24 <joal> Only false positive for Data Loss Warning - Workflow webrequest-load-check_sequence_statistics-wf-upload-2018-4-25-10 [analytics]
2018-04-24 §
16:30 <elukey> restart hadoop hdfs journalnode on analytics1035/52 to pick up prometheus jmx settings [analytics]
14:41 <elukey> restart hadoop hdfs journalnode on analytics1028 to pick up jmx settings [analytics]
12:08 <elukey> restart webrequest-load-wf-text-2018-4-24-9 via Hue (failed due to reimages) [analytics]
06:57 <joal> correct reindextion job: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0033859-180330093100664-oozie-oozi-C/ [analytics]
06:55 <joal> Reindextion job: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0033855-180330093100664-oozie-oozi-C/ [analytics]
06:54 <joal> Manually reindexing all of mediawiki-history for snapshot 2018-03 after having messed it with job testing [analytics]
2018-04-23 §
20:41 <milimetric> deployed a version of wikistats with all but reading metrics disabled to stop showing bad data [analytics]
19:34 <elukey> deploy https://gerrit.wikimedia.org/r/428331 for Pivot [analytics]
14:10 <ottomata> switching main -> analytics MirrorMaker to --new.consumer (temporarily stopping puppet on kafka101[234]) https://phabricator.wikimedia.org/T192387 [analytics]
13:54 <elukey> reimage analytics1067 to debian stretch [analytics]
2018-04-20 §
18:23 <joal> Drop/recreate wmf.mediawiki_user_history andwmf.mediawiki_page_history for T188669 [analytics]
14:17 <elukey> d-[1,2,3] hosts in the analytics labs project upgraded to druid 0.10 [analytics]
11:37 <fdans> manually uploaded refinery whitelist to hdfs [analytics]
11:33 <elukey> reimage analytics1068 do Debian stretch [analytics]
2018-04-19 §
20:39 <milimetric> launched virtual pageviews job, it has id 0026169-180330093100664-oozie-oozi-C [analytics]
20:36 <milimetric> Synced latest refinery version to HDFS [analytics]
17:35 <fdans> refinery deployment - sync to hdfs finished [analytics]
16:27 <elukey> analytics1069 reimaged to Debian stretch [analytics]
15:40 <fdans> deploying refinery [analytics]
14:30 <elukey> disabled druid1001's middlemanager, restarted 1002's [analytics]
14:19 <elukey> add 60G /srv partition to hadoop-coordinator-1 in analytics labs [analytics]
14:04 <elukey> disabled druid1002's worker as prep step for restart - jvms with a old version running realtime indexation [analytics]
2018-04-16 §
10:04 <joal> Restart metrics job after table update [analytics]
09:54 <joal> Update wmf.mediawiki_metrics table for T190058 [analytics]
08:41 <joal> Restart Mediawiki-history job after new patches [analytics]