|
2020-05-07
§
|
| 16:37 |
<elukey> |
roll restart of all the nodemanagers on the hadoop cluster to pick up new jvm settings |
[analytics] |
| 13:53 |
<elukey> |
move stat1007 to role::statistics::explorer (adding jupyterhub) |
[analytics] |
| 11:00 |
<joal> |
Moving application_1583418280867_334532 to the nice queue |
[analytics] |
| 10:58 |
<joal> |
Rerun wikidata-articleplaceholder_metrics-wf-2020-5-6 |
[analytics] |
| 07:45 |
<elukey> |
re-run mediawiki-history-denormalize |
[analytics] |
| 07:43 |
<elukey> |
kill application_1583418280867_333560 after a chat with David, the job is consuming ~2TB of RAM |
[analytics] |
| 07:32 |
<elukey> |
re-run mediawiki history load |
[analytics] |
| 07:18 |
<elukey> |
execute yarn application -movetoqueue application_1583418280867_332862 -queue root.nice |
[analytics] |
| 07:06 |
<elukey> |
restart mediawiki-history-load via hue |
[analytics] |
| 06:41 |
<elukey> |
restart oozie on an-coord1001 |
[analytics] |
| 05:46 |
<elukey> |
re-run mediarequest-hourly-wf-2020-5-6-19 |
[analytics] |
| 05:35 |
<elukey> |
re-run two failed hours for webrequest load text (07/05T05) and upload (06/05T23) |
[analytics] |
| 05:33 |
<elukey> |
restart hadoop yarn nodemanager on analytics1071 |
[analytics] |
|
2020-05-06
§
|
| 12:49 |
<elukey> |
restart oozie on an-coord1001 to pick up the new shlib retention changes |
[analytics] |
| 12:28 |
<mforns> |
re-run pageview-druid-hourly-coord for 2020-05-06T06:00:00 after oozie shared lib update |
[analytics] |
| 11:30 |
<elukey> |
use /run/user as kerberos credential cache for stat1005 |
[analytics] |
| 09:25 |
<elukey> |
re-run projectview coordinator for 2020-5-6-5 after oozie shared lib update |
[analytics] |
| 09:24 |
<elukey> |
re-run virtualpageview coordinator for 2020-5-6-5 after oozie shared lib update |
[analytics] |
| 09:13 |
<elukey> |
re-run apis coordinator for 2020-5-6-7 after oozie shared lib update |
[analytics] |
| 09:11 |
<elukey> |
re-run learning features actor coordinator for 2020-5-6-7 after oozie shared lib update |
[analytics] |
| 09:10 |
<elukey> |
re-run aqs-hourly coordinator for 2020-5-6-7 after oozie shared lib update |
[analytics] |
| 09:09 |
<elukey> |
re-run mediacounts coordinator for 2020-5-6-7 after oozie shared lib update |
[analytics] |
| 09:08 |
<elukey> |
re-run mediarequest coordinator for 2020-5-6-7 after oozie shared lib update |
[analytics] |
| 09:07 |
<elukey> |
re-run data quality coordinators for 2020-5-6-5/6 after oozie shared lib update |
[analytics] |
| 09:05 |
<elukey> |
re-run pageview-hourly coordinator 2020-5-6-6 after oozie shared lib update |
[analytics] |
| 09:04 |
<elukey> |
execute oozie admin -sharelibupdate on an-coord1001 |
[analytics] |
| 06:05 |
<elukey> |
execute hdfs dfs -chown -R analytics-search:analytics-search-users /wmf/data/discovery/search_satisfaction/daily/year=2019 |
[analytics] |
|
2020-05-04
§
|
| 17:08 |
<joal> |
Restart refinery-sqoop-mediawiki-private.service on an-launcher1001 |
[analytics] |
| 17:03 |
<elukey> |
restart refinery-drop-webrequest-refined-partitions after manual chown |
[analytics] |
| 17:03 |
<joal> |
Restart refinery-sqoop-whole-mediawiki.service on an-launcher1001 |
[analytics] |
| 17:02 |
<elukey> |
chown analytics (was: hdfs) /wmf/data/wmf/webrequest/webrequest_source=text/year=2019/month=12/day=14/hour={13,18} |
[analytics] |
| 16:44 |
<joal> |
Deploy refinery again using scap (trying to fox sqoop) |
[analytics] |
| 15:39 |
<joal> |
restart refinery-sqoop-whole-mediawiki.service |
[analytics] |
| 15:37 |
<joal> |
restart refinery-sqoop-mediawiki-private.service |
[analytics] |
| 14:50 |
<joal> |
Deploy refinery using scap to fix sqoop |
[analytics] |
| 13:43 |
<elukey> |
restart refinery-sqoop-whole-mediawiki to test failure exit codes |
[analytics] |
| 06:50 |
<elukey> |
upgrade druid-exporter on all druid nodes |
[analytics] |