1-50 of 885 results (14ms)
2017-07-20 §
19:48 <bd808> Clearing all Eqw state jobs in all queues with: qstat -u '*' | grep Eqw | awk '{print $1;}' | xargs -L1 qmod -cj [tools]
13:54 <andrewbogott> upgrading apache2 on tools-puppetmaster-01 [tools]
04:00 <chasemp> tools-webgrid-lighttpd-1402:~# service nslcd restart && service nscd restart [tools]
03:57 <chasemp> tools-exec-1428:~# service nslcd restart && service nscd restart [tools]
03:57 <bd808> Redtarted cron, nscd, nslcd on tools-cron-01 [tools]
03:45 <chasemp> tools-puppetmaster-01:~# service nslcd restart && service nscd restart [tools]
03:44 <chasemp> tools-puppetmaster-01:~# service nslcd restart && service nscd restart [tools]
03:37 <bd808> Restarted apache on tools-puppetmaster-01 [tools]
2017-07-19 §
23:52 <bd808> Restarted cron on tools-cron-01; toolschecker job showing user not found errors [tools]
21:19 <valhallasw`cloud> Restarted nslcd on tools-bastion-03 (=tools-login); logins seem functional again. [tools]
21:18 <bd808> Forced puppet run and restarted nscd, nslcd on tools-bastion-02 [tools]
2017-07-18 §
19:51 <andrewbogott> enabling puppet on tools-proxy-02. I don't know why it was disabled. [tools]
2017-07-17 §
01:43 <bd808> Uncordoned tools-worker-1020 after it deleted pods with local storage that were filling the entire disk [tools]
01:36 <bd808> Depooling tools-worker-1020 [tools]
2017-07-13 §
21:59 <bd808> Elasticsearch cluster upgraded to 5.3.2 [tools]
21:25 <bd808> Upgrading ElasticSearch cluster for T164842. There will be service interruptions [tools]
17:59 <bd808> Puppet is disabled on tools-proxy-02 with no reason specified. [tools]
17:09 <bd808> Upgraded nginx-common on tools-proxy-02 [tools]
17:05 <bd808> Upgraded nginx-common on tools-proxy-01 [tools]
2017-07-12 §
15:46 <chasemp> push out puppet run across tools [tools]
12:15 <andrewbogott> restarting 'admin' webservice [tools]
2017-07-07 §
18:26 <bd808> Forced puppet runs on tools-redis-* for security fix [tools]
2017-07-03 §
04:26 <bd808> cdnjs on tools-static-10 is up to date [tools]
03:38 <bd808> cdnjs on tools-static-11 is up to date [tools]
02:19 <bd808> Cleaning up stuck merges for cdnjs clones on tools-static-10 and tools-static-11 [tools]
2017-07-01 §
19:40 <bd808> Disabled puppet on tools-k8s-master-01 to try and fix maintain-kubeusers [tools]
19:32 <bd808> Restarted maintain-kubeusers on tools-k8s-master-01 [tools]
2017-06-30 §
01:33 <chasemp> time for i in `cat tools-hosts`; do ssh -i ~/.ssh/labs_root_id_rsa root@$i.eqiad.wmflabs 'hostname -f; uptime; tc-setup'; done [tools]
01:29 <andrewbogott> rebooting tools-cron-01 [tools]
2017-06-29 §
23:01 <madhuvishy> Uncordoned all k8s-workers [tools]
20:50 <madhuvishy> deppoling, rebooting and repooling all grid exec nodes [tools]
20:33 <andrewbogott> depooling, rebooting, and repooling every lighttpd node three at a time [tools]
19:55 <madhuvishy> Killed liangent-php jobs and usrd-tools jobs [tools]
18:00 <madhuvishy> drain cordon reboot uncordon tools-worker-1015 [tools]
17:37 <madhuvishy> drain cordon reboot uncordon tools-worker-1005 tools-worker-1007 tools-worker-1008 [tools]
17:22 <bd808> rebooting tools-static-11 [tools]
17:20 <andrewbogott> rebooting tools-static-10 [tools]
17:20 <madhuvishy> drain cordon reboot uncordon tools-worker-1012 tools-worker-1003 [tools]
17:13 <madhuvishy> drain cordon reboot uncordon tools-worker-1022, tools-worker-1009, tools-worker-1002 [tools]
16:27 <chasemp> restart k8s components on master (madhu) [tools]
16:10 <chasemp> tools-flannel-etcd-01:~$ sudo service etcd restart [tools]
16:04 <madhuvishy> reboot tools-worker-1022 tools-worker-1009 [tools]
15:57 <chasemp> reboot tools-docker-registery-01 for nfs [tools]
2017-06-27 §
21:32 <andrewbogott> moving all tools nodes to new puppetmaster, tools-puppetmaster-01.tools.eqiad.wmflabs [tools]
2017-06-25 §
15:13 <madhuvishy> Restarted webservice on tools.fatameh [tools]
2017-06-24 §
16:01 <bd808> Created and provisioned elasticsearch password for tools.wmde-uca-test (T167971) [tools]
2017-06-23 §
20:20 <bd808> Reindexing various elasticsearch indexes created before we upgraded to v2.x [tools]
20:19 <bd808> Dropped garbage indexes in elasticsearch cluster [tools]
2017-06-22 §
17:03 <bd808> Rolled back attempt at Elasticsearch upgrade. Indices need to be rebuilt with 2.x before 5.x can be installed. T164842 [tools]
16:19 <bd808> Backed up elasticsearch indexes to personal laptop using elasticdump incase T164842 goes horribly wrong [tools]