1-50 of 895 results (9ms)
2017-08-09 §
17:28 <chasemp> webservices restart tools.orphantalk [tools]
2017-08-03 §
00:47 <bd808> tools-bastion-03 not usably responsive to interactive commands; will reboot [tools]
00:00 <bd808> Restarted kube-proxy service on bastion-03 [tools]
2017-08-02 §
16:59 <bd808> Force deleted 6 jobs suck in 'dr' state [tools]
2017-07-31 §
15:28 <chasemp> remove python-keystoneclient from bastion-03 [tools]
2017-07-27 §
23:27 <bd808> Killed python procs owned by sdesabbata on tools-login that were stealing all cpu/io [tools]
21:16 <bd808> Disabled puppet on tools-proxy-01 to test nginx proxy config changes [tools]
16:27 <bd808> Enabled puppet on tools-static-11 [tools]
16:10 <bd808> Disabled puppet on tools-static-11 to test https://gerrit.wikimedia.org/r/#/c/357878 [tools]
2017-07-26 §
22:33 <chasemp> hotpatching an hiera value on tools master to see effects [tools]
2017-07-20 §
19:48 <bd808> Clearing all Eqw state jobs in all queues with: qstat -u '*' | grep Eqw | awk '{print $1;}' | xargs -L1 qmod -cj [tools]
13:54 <andrewbogott> upgrading apache2 on tools-puppetmaster-01 [tools]
04:00 <chasemp> tools-webgrid-lighttpd-1402:~# service nslcd restart && service nscd restart [tools]
03:57 <chasemp> tools-exec-1428:~# service nslcd restart && service nscd restart [tools]
03:57 <bd808> Redtarted cron, nscd, nslcd on tools-cron-01 [tools]
03:45 <chasemp> tools-puppetmaster-01:~# service nslcd restart && service nscd restart [tools]
03:44 <chasemp> tools-puppetmaster-01:~# service nslcd restart && service nscd restart [tools]
03:37 <bd808> Restarted apache on tools-puppetmaster-01 [tools]
2017-07-19 §
23:52 <bd808> Restarted cron on tools-cron-01; toolschecker job showing user not found errors [tools]
21:19 <valhallasw`cloud> Restarted nslcd on tools-bastion-03 (=tools-login); logins seem functional again. [tools]
21:18 <bd808> Forced puppet run and restarted nscd, nslcd on tools-bastion-02 [tools]
2017-07-18 §
19:51 <andrewbogott> enabling puppet on tools-proxy-02. I don't know why it was disabled. [tools]
2017-07-17 §
01:43 <bd808> Uncordoned tools-worker-1020 after it deleted pods with local storage that were filling the entire disk [tools]
01:36 <bd808> Depooling tools-worker-1020 [tools]
2017-07-13 §
21:59 <bd808> Elasticsearch cluster upgraded to 5.3.2 [tools]
21:25 <bd808> Upgrading ElasticSearch cluster for T164842. There will be service interruptions [tools]
17:59 <bd808> Puppet is disabled on tools-proxy-02 with no reason specified. [tools]
17:09 <bd808> Upgraded nginx-common on tools-proxy-02 [tools]
17:05 <bd808> Upgraded nginx-common on tools-proxy-01 [tools]
2017-07-12 §
15:46 <chasemp> push out puppet run across tools [tools]
12:15 <andrewbogott> restarting 'admin' webservice [tools]
2017-07-07 §
18:26 <bd808> Forced puppet runs on tools-redis-* for security fix [tools]
2017-07-03 §
04:26 <bd808> cdnjs on tools-static-10 is up to date [tools]
03:38 <bd808> cdnjs on tools-static-11 is up to date [tools]
02:19 <bd808> Cleaning up stuck merges for cdnjs clones on tools-static-10 and tools-static-11 [tools]
2017-07-01 §
19:40 <bd808> Disabled puppet on tools-k8s-master-01 to try and fix maintain-kubeusers [tools]
19:32 <bd808> Restarted maintain-kubeusers on tools-k8s-master-01 [tools]
2017-06-30 §
01:33 <chasemp> time for i in `cat tools-hosts`; do ssh -i ~/.ssh/labs_root_id_rsa root@$i.eqiad.wmflabs 'hostname -f; uptime; tc-setup'; done [tools]
01:29 <andrewbogott> rebooting tools-cron-01 [tools]
2017-06-29 §
23:01 <madhuvishy> Uncordoned all k8s-workers [tools]
20:50 <madhuvishy> deppoling, rebooting and repooling all grid exec nodes [tools]
20:33 <andrewbogott> depooling, rebooting, and repooling every lighttpd node three at a time [tools]
19:55 <madhuvishy> Killed liangent-php jobs and usrd-tools jobs [tools]
18:00 <madhuvishy> drain cordon reboot uncordon tools-worker-1015 [tools]
17:37 <madhuvishy> drain cordon reboot uncordon tools-worker-1005 tools-worker-1007 tools-worker-1008 [tools]
17:22 <bd808> rebooting tools-static-11 [tools]
17:20 <andrewbogott> rebooting tools-static-10 [tools]
17:20 <madhuvishy> drain cordon reboot uncordon tools-worker-1012 tools-worker-1003 [tools]
17:13 <madhuvishy> drain cordon reboot uncordon tools-worker-1022, tools-worker-1009, tools-worker-1002 [tools]
16:27 <chasemp> restart k8s components on master (madhu) [tools]