951-1000 of 4851 results (26ms)
2021-06-15 §
16:33 <elukey> restart hadoop-yarn-resourcemanager on an-master1001 [analytics]
16:16 <razzi> sudo systemctl stop 'hadoop-*' on an-master1002 [analytics]
16:14 <razzi> sudo systemctl stop hadoop-* on an-master1001, then realize I meant to do this on an-master1002, so start hadoop-* [analytics]
16:11 <razzi> downtime an-master1002 [analytics]
15:55 <razzi> sudo transfer.py an-master1001.eqiad.wmnet:/srv/hadoop/backup/hdfs-namenode-snapshot-buster-reimage-2021-06-15.tar.gz stat1004.eqiad.wmnet:/home/razzi/hdfs-namenode-fsimage [analytics]
15:42 <razzi> tar -czf /srv/hadoop/backup/hdfs-namenode-snapshot-buster-reimage-$(date --iso-8601).tar.gz current on an-master1001 [analytics]
15:38 <razzi> backup /srv/hadoop/name/current to /home/razzi/hdfs-namenode-snapshot-buster-reimage-2021-06-15.tar.gz on an-master1001 [analytics]
15:33 <razzi> sudo -u hdfs kerberos-run-command hdfs hdfs dfsadmin -saveNamespace [analytics]
15:27 <razzi> sudo -u hdfs kerberos-run-command hdfs hdfs dfsadmin -safemode enter [analytics]
15:25 <razzi> kill running yarn applications via for loop [analytics]
15:11 <razzi> sudo -u yarn kerberos-run-command yarn yarn rmadmin -refreshQueues [analytics]
15:09 <razzi> disable puppet on an-mastesr [analytics]
15:08 <razzi> run puppet on an-masters to update capacity-scheduler.xml [analytics]
15:02 <razzi> disable puppet on an-masters [analytics]
15:01 <razzi> sudo -u yarn kerberos-run-command yarn yarn rmadmin -refreshQueues to stop queues [analytics]
14:35 <razzi> disable jobs that use hadoop on an-launcher1002 following https://phabricator.wikimedia.org/T278423#7094641 [analytics]
2021-06-14 §
18:45 <ottomata> remove packges from hadoop common nodes: sudo cumin 'R:Class = profile::analytics::cluster::packages::common' 'apt-get -y remove python3-pandas python3-pycountry python3-numpy python3-tz' - T275786 [analytics]
18:43 <ottomata> remove packges from stat nodes: sudo cumin 'stat*' apt-get -y remove subversion mercurial tofrodos libwww-perl libcgi-pm-perl libjson-perl libtext-csv-xs-perl libproj-dev libboost-regex-dev libboost-system-dev libgoogle-glog-dev libboost-iostreams-dev libgdal-dev [analytics]
07:18 <joal> Rerun cassandra-daily-wf-local_group_default_T_pageviews_per_article_flat-2021-6-11 [analytics]
2021-06-10 §
21:17 <razzi> sudo systemctl restart monitor_refine_eventlogging_analytics [analytics]
18:17 <razzi> sudo systemctl restart hadoop-mapreduce-historyserver [analytics]
17:24 <razzi> sudo systemctl restart hadoop-hdfs-namenode on an-master1002 [analytics]
17:24 <razzi> sudo systemctl restart hadoop-hdfs-zkfc on an-master1002 [analytics]
17:12 <razzi> sudo -u hdfs /usr/bin/hdfs haadmin -failover an-master1002-eqiad-wmnet an-master1001-eqiad-wmnet [analytics]
16:25 <razzi> rolling restart hadoop masters to pick up https://gerrit.wikimedia.org/r/c/operations/puppet/+/698194 [analytics]
14:07 <ottomata> altered event.wmdebannerevent event.eventRate field to change type from BIGINT to DOUBLE - T282562 [analytics]
2021-06-08 §
16:56 <elukey> move away from dbstore1004 in favor of dbstore1007 in analytics CNAME/SRV records (will affect analytics-mysql and sqoop) [analytics]
13:42 <ottomata> roll restart an-conf zookeepers - T283067 [analytics]
13:22 <ottomata> roll restarting analytics presto-servers - T283067 [analytics]
06:08 <elukey> restart yarn nodemanager on analytics1075 to clear the un-healthy state after some days of downtime (one-off issue but let's keep an eye on it) [analytics]
2021-06-07 §
18:14 <ottomata> rolling restart of kafka jumbo brokers - T283067 [analytics]
17:53 <ottomata> rolling restart of kafka jumbo mirror makers - T283067 [analytics]
17:07 <ottomata> remove packages from an clsuter nodes: sudo apt-get -y remove r-cran-rmysql python3-matplotlib python3-sklearn python3-enchant python3-nltk gfortran liblapack-dev libopenblas-dev - T275786 [analytics]
16:50 <ottomata> restarting mysqld analytics-meta replica on db1108 to apply config change - T272973 [analytics]
2021-06-04 §
17:42 <razzi> sudo cookbook sre.aqs.roll-restart aqs to deploy new mediawiki history snapshot [analytics]
2021-06-03 §
22:32 <razzi> sudo manage_principals.py create jdl --email_address=jlinehan@wikimedia.org [analytics]
22:32 <razzi> sudo manage_principals.py create phuedx --email_address=phuedx@wikimedia.org [analytics]
15:46 <ottomata> add airflow_2.1.0-py3.7-1_amd64.deb to apt.wm.org [analytics]
15:20 <ottomata> created airflow_analytics database and user on an-coord1001 analytics-meta instance - T272973 [analytics]
2021-06-02 §
18:09 <ottomata> remove .deb packages from stat boxes: python3-mysqldb python3-boto python3-ua-parser python3-netaddr python3-pymysql python3-protobuf python3-unidecode python3-oauth2client python3-oauthlib python3-requests-oauthlib python3-ua-parser - T275786 [analytics]
2021-05-31 §
06:56 <joal> Rerun cassandra-daily-wf-local_group_default_T_pageviews_per_article_flat-2021-5-29 [analytics]
2021-05-27 §
14:37 <elukey> removed Luca's and Tobias' emails from analytics-alerts@ [analytics]
07:01 <elukey> roll restart hdfs namenodes to pick up new GC/heap settings - https://gerrit.wikimedia.org/r/c/operations/puppet/+/695933 [analytics]
2021-05-26 §
19:14 <ottomata> deploying refinery and refinery source 0.1.13 [analytics]
17:29 <ottomata> killing and restarting oozie cassandra loader jobs coord_unique_devices_daily and coord_pageview_top_percountry_daily after revert of oozie job to load to cassandra 3 [analytics]
14:18 <ottomata> deploying refinery... [analytics]
14:17 <ottomata> Deployed refinery-source using jenkins [analytics]
2021-05-25 §
18:16 <razzi> sudo systemctl start all failed units from `systemctl list-units --state=failed` on an-launcher1002 [analytics]
18:14 <razzi> sudo systemctl start eventlogging_to_druid_navigationtiming_hourly.service [analytics]
18:01 <razzi> manually edit /etc/hadoop/conf/capacity-scheduler.xml to make queues running and sudo -u yarn kerberos-run-command yarn yarn rmadmin -refreshQueues [analytics]