1051-1100 of 2017 results (14ms)
|
2017-12-01
§
|
| 15:33 |
<chasemp> |
put the weird mess of untracked files on tools puppetmaster into stash to see what breaks as they should not be there? |
[tools] |
| 15:30 |
<chasemp> |
prometheus nfs collector on tools-bastion-03 |
[tools] |
|
2017-11-30
§
|
| 23:23 |
<bd808> |
Hard reboot of tools-bastion-03 via Horizon |
[tools] |
| 23:06 |
<chasemp> |
rebooting login.tools.wmflabs.org due to overload |
[tools] |
|
2017-11-20
§
|
| 20:34 |
<chasemp> |
backup crons tools-cron-01:/var/spool/cron# cp -Rp crontabs/ /root/20112017/ |
[tools] |
| 00:52 |
<andrewbogott> |
cherry-picking https://gerrit.wikimedia.org/r/#/c/392172/ onto the tools puppetmaster |
[tools] |
|
2017-11-17
§
|
| 21:33 |
<valhallasw`cloud> |
also g-w'ed those files, and sent emails to all the affected users |
[tools] |
| 21:17 |
<valhallasw`cloud> |
chmod o-w'ed a bunch of files reported by Dispenser; writing emails to the owners about this |
[tools] |
|
2017-11-16
§
|
| 17:40 |
<chasemp> |
tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --enable && sudo puppet agent --test && sudo unattended-upgrades -d' |
[tools] |
| 16:50 |
<bd808> |
Force upgraded nginx on tools-elastic-* |
[tools] |
| 16:37 |
<chasemp> |
reboot tools-checker-01 |
[tools] |
| 15:17 |
<chasemp> |
disable puppet |
[tools] |
|
2017-11-15
§
|
| 22:48 |
<madhuvishy> |
Rebooted tools-paws-worker-1017 |
[tools] |
| 15:53 |
<chasemp> |
reboot bastion-03 |
[tools] |
| 15:48 |
<chasemp> |
kill tools.powow on bastion-03 for hammering IO and making bastion unusable |
[tools] |
|
2017-11-07
§
|
| 01:21 |
<bd808> |
Removed all non-directory files from /home (via labstore1004 direct access) |
[tools] |
|
2017-11-06
§
|
| 18:30 |
<bd808> |
Load on tools-bastion-03 down to 0.72 from 17.47 after killing a bunch of local processes that should have been running on the job grid instead |
[tools] |
|
2017-11-05
§
|
| 23:48 |
<bd808> |
Cleaned up 2 huge /tmp files left by tools.croptool (~6.5G) |
[tools] |
| 23:44 |
<bd808> |
Cleaned up 109 files owned by tools.rezabot on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.rezabot -exec rm {} \+` |
[tools] |
| 23:37 |
<bd808> |
Cleaned up 955 files owned by tools.wsexport on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.wsexport -exec rm {} \+` |
[tools] |
|
2017-11-03
§
|
| 21:19 |
<bd808> |
Deployed misctools 1.26 (T156174) |
[tools] |
|
2017-11-02
§
|
| 16:15 |
<bd808> |
Restarted nslcd on tools-bastion-03 |
[tools] |
|
2017-11-01
§
|
| 07:11 |
<madhuvishy> |
Clear nscd cache across all projects post labsdb dns switchover T179464 |
[tools] |
| 07:11 |
<madhuvishy> |
Clear nscd cache across all projects post labsdb dns switchover |
[tools] |
|
2017-10-31
§
|
| 16:50 |
<bd808> |
tools-bastion-03 (tools-login, login.tools) is overloaded |
[tools] |
|
2017-10-30
§
|
| 17:35 |
<madhuvishy> |
Clear dns caches across tools hosts `sudo nscd -i hosts` |
[tools] |
| 16:08 |
<arturo> |
depool tools-exec-1401.tools.eqiad.wmflabs |
[tools] |
|
2017-10-24
§
|
| 18:09 |
<madhuvishy> |
Disable puppet on tools-package-builder-01 temporarily (T178920) |
[tools] |
| 13:22 |
<chasemp> |
start admin webservice |
[tools] |
| 13:22 |
<chasemp> |
stop admin webservice |
[tools] |
|
2017-10-23
§
|
| 14:49 |
<chasemp> |
wall message and scheduled reboot in 5m for bastion-03 |
[tools] |
|
2017-10-18
§
|
| 21:36 |
<chasemp> |
stop basebot -- it is going crazy and spamming email w/ failing to log to error.log. Need to figure out how to notify but it's clearly in a failure loop. |
[tools] |
| 14:04 |
<chasemp> |
add strephit creds to elasticsearch per T178310 |
[tools] |
|
2017-10-12
§
|
| 16:57 |
<bd808> |
Rebuilding all Kubernetes Docker images to include toollabs-webservice 0.38 |
[tools] |
| 16:53 |
<bd808> |
Upgraded toollabs-webservice to 0.38 |
[tools] |
|
2017-10-06
§
|
| 15:33 |
<bd808> |
Upgrade jobutils to 1.25 (T177614) |
[tools] |
| 00:27 |
<bd808> |
Updated misctools to 1.24 |
[tools] |
|
2017-10-05
§
|
| 22:47 |
<bd808> |
Updated misctools to 1.23 |
[tools] |
| 22:42 |
<bd808> |
Updated jobutils to 1.23 |
[tools] |
| 15:46 |
<chasemp> |
tools-bastion-03 has tons of local tools running long lived NFS intensive processes. I'm rebooting rather than playing whackamole. |
[tools] |
|
2017-10-03
§
|
| 19:30 |
<bd808> |
`kubectl --namespace=prod delete pod --all` on tools-paws-master-01 |
[tools] |
|
2017-10-01
§
|
| 21:46 |
<madhuvishy> |
Cold migrating tools-clushmaster-01 from labvirt1015 to labvirt1017 |
[tools] |
|
2017-09-29
§
|
| 19:49 |
<andrewbogott> |
migration tools-clushmaster-01 to labvirt1015 |
[tools] |
|
2017-09-25
§
|
| 15:14 |
<andrewbogott> |
rebooting tools-paws-worker-1006 since I can't access it |
[tools] |
| 14:57 |
<chasemp> |
OS_TENANT_NAME=tools openstack server reboot 2c0cf363-c7c3-42ad-94bd-e586f2492321 (unresponsive) |
[tools] |
|
2017-09-20
§
|
| 16:52 |
<madhuvishy> |
apt-get install --only-upgrade apache2; service apache2 restart on tools-puppetmaster-01 |
[tools] |
|
2017-09-19
§
|
| 15:22 |
<chasemp> |
tools-clushmaster-01:~$ clush -f 5 -g all 'sudo puppet agent --test' |
[tools] |
| 13:39 |
<chasemp> |
bastion-03 someone dropped 8.6G in /tmp which is /not/ seemingly on a temp file system |
[tools] |
| 13:25 |
<chasemp> |
wall Bastion disk is full and needs attention and reboot in 60 |
[tools] |
|
2017-09-18
§
|
| 18:02 |
<bd808> |
Updated PHP5.6 images for Kubernetes (T172358) |
[tools] |