701-750 of 943 results (16ms)
|
2017-08-02
§
|
| 16:59 |
<bd808> |
Force deleted 6 jobs suck in 'dr' state |
[tools] |
|
2017-07-27
§
|
| 23:27 |
<bd808> |
Killed python procs owned by sdesabbata on tools-login that were stealing all cpu/io |
[tools] |
| 21:16 |
<bd808> |
Disabled puppet on tools-proxy-01 to test nginx proxy config changes |
[tools] |
| 16:27 |
<bd808> |
Enabled puppet on tools-static-11 |
[tools] |
| 16:10 |
<bd808> |
Disabled puppet on tools-static-11 to test https://gerrit.wikimedia.org/r/#/c/357878 |
[tools] |
|
2017-07-20
§
|
| 19:48 |
<bd808> |
Clearing all Eqw state jobs in all queues with: qstat -u '*' | grep Eqw | awk '{print $1;}' | xargs -L1 qmod -cj |
[tools] |
| 03:57 |
<bd808> |
Redtarted cron, nscd, nslcd on tools-cron-01 |
[tools] |
| 03:37 |
<bd808> |
Restarted apache on tools-puppetmaster-01 |
[tools] |
|
2017-07-19
§
|
| 23:52 |
<bd808> |
Restarted cron on tools-cron-01; toolschecker job showing user not found errors |
[tools] |
| 21:18 |
<bd808> |
Forced puppet run and restarted nscd, nslcd on tools-bastion-02 |
[tools] |
|
2017-07-17
§
|
| 01:43 |
<bd808> |
Uncordoned tools-worker-1020 after it deleted pods with local storage that were filling the entire disk |
[tools] |
| 01:36 |
<bd808> |
Depooling tools-worker-1020 |
[tools] |
|
2017-07-13
§
|
| 21:59 |
<bd808> |
Elasticsearch cluster upgraded to 5.3.2 |
[tools] |
| 21:25 |
<bd808> |
Upgrading ElasticSearch cluster for T164842. There will be service interruptions |
[tools] |
| 17:59 |
<bd808> |
Puppet is disabled on tools-proxy-02 with no reason specified. |
[tools] |
| 17:09 |
<bd808> |
Upgraded nginx-common on tools-proxy-02 |
[tools] |
| 17:05 |
<bd808> |
Upgraded nginx-common on tools-proxy-01 |
[tools] |
|
2017-07-07
§
|
| 18:26 |
<bd808> |
Forced puppet runs on tools-redis-* for security fix |
[tools] |
|
2017-07-03
§
|
| 04:26 |
<bd808> |
cdnjs on tools-static-10 is up to date |
[tools] |
| 03:38 |
<bd808> |
cdnjs on tools-static-11 is up to date |
[tools] |
| 02:19 |
<bd808> |
Cleaning up stuck merges for cdnjs clones on tools-static-10 and tools-static-11 |
[tools] |
|
2017-07-01
§
|
| 19:40 |
<bd808> |
Disabled puppet on tools-k8s-master-01 to try and fix maintain-kubeusers |
[tools] |
| 19:32 |
<bd808> |
Restarted maintain-kubeusers on tools-k8s-master-01 |
[tools] |
|
2017-06-29
§
|
| 20:50 |
<madhuvishy> |
deppoling, rebooting and repooling all grid exec nodes |
[tools] |
| 20:33 |
<andrewbogott> |
depooling, rebooting, and repooling every lighttpd node three at a time |
[tools] |
| 19:55 |
<madhuvishy> |
Killed liangent-php jobs and usrd-tools jobs |
[tools] |
| 17:22 |
<bd808> |
rebooting tools-static-11 |
[tools] |
|
2017-06-24
§
|
| 16:01 |
<bd808> |
Created and provisioned elasticsearch password for tools.wmde-uca-test (T167971) |
[tools] |
|
2017-06-23
§
|
| 20:20 |
<bd808> |
Reindexing various elasticsearch indexes created before we upgraded to v2.x |
[tools] |
| 20:19 |
<bd808> |
Dropped garbage indexes in elasticsearch cluster |
[tools] |
|
2017-06-22
§
|
| 17:03 |
<bd808> |
Rolled back attempt at Elasticsearch upgrade. Indices need to be rebuilt with 2.x before 5.x can be installed. T164842 |
[tools] |
| 16:19 |
<bd808> |
Backed up elasticsearch indexes to personal laptop using elasticdump incase T164842 goes horribly wrong |
[tools] |
| 00:12 |
<bd808> |
Set ownership and permissions on $HOME/.kube for all tools (T165875) |
[tools] |
|
2017-06-14
§
|
| 22:09 |
<bd808> |
Restarted apache2 proc on tools-puppetmaster-02 |
[tools] |
|
2017-06-08
§
|
| 18:14 |
<madhuvishy> |
Also delete from /tmp on tools-webgrid-lighttpd-1411 xvfb-run.*, calibre_* and ws-*.epub |
[tools] |
| 18:07 |
<madhuvishy> |
Clean up space on /tmp on tools-webgrid-lighttpd-1426 by deleting temp files xvfb-run.* and calibre_1.25.0_tmp_* created by the wsexport tool |
[tools] |
|
2017-06-06
§
|
| 20:30 |
<chasemp> |
rebooting tools-bastion-02 as unresponsive (up 76 days and lots of seemingly left behind things running) |
[tools] |
|
2017-06-05
§
|
| 23:44 |
<bd808> |
Deleted tools.iabot crontab that somehow got locally installed on tools-exec-1412 on 2017-05-24T20:55Z |
[tools] |
| 22:15 |
<bd808> |
Deleted tools.aibot crontab that somehow got locally installed on tools-exec-1436 on 2017-05-24T20:55Z |
[tools] |
| 19:55 |
<andrewbogott> |
disabling puppet on tools-proxy-01 and -02 for a staged rollout of https://gerrit.wikimedia.org/r/#/c/350494/16 |
[tools] |
|
2017-05-31
§
|
| 19:29 |
<bd808> |
Rebuiding all Docker images to pick up toollabs-webservice v0.37 (T163355) |
[tools] |
| 19:24 |
<bd808> |
Updating toolabs-webservice package via clush (T163355) |
[tools] |
| 19:16 |
<bd808> |
Installed toollabs-webservice_0.37_all.deb from local file on tools-bastion-02 (T163355) |
[tools] |
|
2017-05-30
§
|
| 17:15 |
<madhuvishy> |
Drained and rebooted tools-worker-1006 |
[tools] |
| 17:15 |
<madhuvishy> |
depooled, rebooted, and repooled tools-exec-1412 |
[tools] |
|
2017-05-26
§
|
| 20:32 |
<bd808> |
Added tools-webgrid-lighttpd-14{19,2[0-8]} as submit hosts |
[tools] |
| 20:31 |
<bd808> |
Added tools-webgrid-lighttpd-1412 and tools-webgrid-lighttpd-1413 as submit hosts |
[tools] |
| 20:28 |
<bd808> |
sudo qconf -as tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs |
[tools] |
|
2017-05-10
§
|
| 19:11 |
<bd808> |
Edited striker db record for user Stepan Grigoryev to detach SUL and Phab accounts. T164849 |
[tools] |
| 17:47 |
<bd808> |
Signed and revoked puppet certs generated when our DNS flipped out and gave hosts non-FQDN hostnames |
[tools] |