2001-2050 of 4824 results (27ms)
2020-07-03 §
18:51 <joal> restart virtualpageview-hourly-wf-2020-7-3-15 after hive-server failure [analytics]
16:41 <joal> Rerun mediawiki-history-check_denormalize-wf-2020-06 after having cleaned up wrong files and restarted a job without deterministic skewed join [analytics]
2020-07-02 §
18:16 <joal> Launch a manual instance of mediawiki-history-denormalize to release data despite oozie failing [analytics]
16:17 <joal> rerun mediawiki-history-denormalize-wf-2020-06 after oozie sharelib bump through manual restart [analytics]
12:41 <joal> retry mediawiki-history-denormalize-wf-2020-06 [analytics]
07:26 <elukey> start a tmux on an-launcher1002 with 'sudo -u analytics /usr/local/bin/kerberos-run-command analytics /usr/local/bin/refinery-sqoop-mediawiki-production' [analytics]
07:20 <elukey> execute systemctl reset-failed refinery-sqoop-whole-mediawiki.service to clear our alarms on launcher1002 [analytics]
2020-07-01 §
19:04 <joal> Kill/restart webrequest-load-bundle for mobile-pageview update [analytics]
18:59 <joal> kill/restart pageview-druid jobs (hourly, daily, monthly) for in_content_namespace field update [analytics]
18:57 <joal> kill/restart mediawiki-wikitext-history-coord and mediawiki-wikitext-current-coord for bz2 codec update [analytics]
18:55 <joal> kill/restart mediawiki-history-denormalize-coord after skewed-join strategy update [analytics]
18:52 <joal> Kill/Restart unique_devices-per_project_family-monthly-coord after fix [analytics]
18:41 <joal> deploy refinery to HDFS [analytics]
18:28 <joal> Deploy refinery using scap after hotfix [analytics]
18:20 <joal> Deploy refinery using scap [analytics]
16:58 <joal> trying to release refinery-source 0.0.129 to archiva, version 3 [analytics]
16:51 <elukey> remove /etc/maven/settings.xml from all analytics nodes that have it [analytics]
2020-06-30 §
18:28 <joal> trying to release refinery-source to archiva from jenkins (second time) [analytics]
16:30 <joal> Release refinery-source v0.0.129 using jenkins [analytics]
16:30 <joal> Deploy refien [analytics]
16:05 <elukey> re-enable timers on an-launcher1002 after archiva maintenance [analytics]
15:23 <elukey> stop timers on an-launcher1002 to ease debugging for refinery deploy [analytics]
13:12 <elukey> restart nodemanager on analytics1068 after GC overhead and OOMs [analytics]
09:32 <joal> Kill/Restart mediawiki-wikitext-history job now that the current month one is done (bz2 fix) [analytics]
2020-06-29 §
13:09 <elukey> archiva.wikimedia.org migrated to archiva1002 [analytics]
2020-06-25 §
17:20 <elukey> move RU jobs/timers from an-launcher1001 to an-launcher1002 [analytics]
16:07 <elukey> move all timers but RU from an-launcher1001 to 1002 (puppet disabled on 1001, all timers completed) [analytics]
12:13 <elukey> reimage notebook1003/4 to debian buster as fresh start [analytics]
09:28 <joal> Kill-restart pageview-hourly to read from pageview_actor [analytics]
09:25 <joal> Kill-restart pageview_actor jobs (current+backfill) after dpeloy [analytics]
09:14 <joal> Deploy refinery to HDFS [analytics]
08:56 <joal> deploying refinery using scap to fix pageview_actor_hourly [analytics]
08:02 <joal> Start backfilling pageview_actor_hourly job with new patch (expected to solve heisenbug) [analytics]
07:40 <joal> Dropping refinery-camus jars from archiva up to 0.0.115 [analytics]
07:04 <joal> rerun failed pageview_actor_hourly [analytics]
2020-06-24 §
19:36 <joal> Cleaning refinery-spark from archiva (up to 0.0.115) [analytics]
19:28 <joal> Cleaning refinery-tools from archiva (up to 0.0.115) [analytics]
19:16 <joal> Restarting unique-devices jobs to use pageview_actor_hourly instead of webrequest (4 jobs) [analytics]
19:08 <joal> Start pageview_actor_hourly oozie job [analytics]
19:06 <joal> Create pageview_actor_hourly after deploy to start new jobs [analytics]
18:57 <joal> Clean archiva refinery-camus except 0.0.90 [analytics]
18:54 <joal> Deploying refinery onto HDFS [analytics]
18:47 <joal> clean archiva from refinery-hive (up to 0.0.115) [analytics]
18:47 <joal> Deploying refinery using scap [analytics]
18:15 <joal> launching a new jenkins release after cleanup [analytics]
17:43 <joal> Reseting refinery-source to v0.0.128 for clean release after jenkins-archiva password fix [analytics]
16:20 <joal> Releasing refinery-source 0.0.128 to archiva [analytics]
06:50 <elukey> truncate /srv/reportupdater/log/reportupdater-ee-beta-features from 43G to 1G on an-launcher1001 (disk space issues) [analytics]
2020-06-22 §
18:50 <joal> Manually update pageview whitelist adding shnwiktionary [analytics]
2020-06-20 §
07:41 <elukey> powercycle an-worker1093 - bug soft lock up CPU showed in mgmt console [analytics]