|
2018-01-16
§
|
| 21:54 |
<chasemp> |
tools-exec-1436:~$ /sbin/reboot |
[tools] |
| 21:24 |
<andrewbogott> |
repooled tools-exec-1420 and tools-webgrid-lighttpd-1417 |
[tools] |
| 21:14 |
<andrewbogott> |
depooling tools-exec-1420 and tools-webgrid-lighttpd-1417 |
[tools] |
| 20:58 |
<andrewbogott> |
depooling tools-exec-1412, 1415, 1417, tools-webgrid-lighttpd-1415, 1416, 1422, 1426 |
[tools] |
| 20:56 |
<andrewbogott> |
repooling tools-exec-1416, 1418, 1424, tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1410 |
[tools] |
| 20:46 |
<andrewbogott> |
depooling tools-exec-1416, 1418, 1424, tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1410 |
[tools] |
| 20:46 |
<andrewbogott> |
repooled tools-exec-1406, 1421, 1436, 1437, tools-webgrid-generic-1404, tools-webgrid-lighttpd-1409, tools-webgrid-lighttpd-1411, tools-webgrid-lighttpd-1418, tools-webgrid-lighttpd-1425 |
[tools] |
| 20:33 |
<andrewbogott> |
depooling tools-exec-1406, 1421, 1436, 1437, tools-webgrid-generic-1404, tools-webgrid-lighttpd-1409, tools-webgrid-lighttpd-1411, tools-webgrid-lighttpd-1418, tools-webgrid-lighttpd-1425 |
[tools] |
| 20:20 |
<andrewbogott> |
depooling tools-webgrid-lighttpd-1412 and tools-exec-1423 for host reboot |
[tools] |
| 20:19 |
<andrewbogott> |
repooled tools-exec-1409, 1410, 1414, 1419, 1427, 1428 tools-webgrid-generic-1401, tools-webgrid-lighttpd-1406 |
[tools] |
| 20:02 |
<andrewbogott> |
depooling tools-exec-1409, 1410, 1414, 1419, 1427, 1428 tools-webgrid-generic-1401, tools-webgrid-lighttpd-1406 |
[tools] |
| 20:00 |
<andrewbogott> |
depooled and repooled tools-webgrid-lighttpd-1427 tools-webgrid-lighttpd-1428 tools-exec-1413 tools-exec-1442 for host reboot |
[tools] |
| 18:50 |
<andrewbogott> |
switched active proxy back to tools-proxy-02 |
[tools] |
| 18:50 |
<andrewbogott> |
repooling tools-exec-1422 and tools-webgrid-lighttpd-1413 |
[tools] |
| 18:34 |
<andrewbogott> |
moving proxy from tools-proxy-02 to tools-proxy-01 |
[tools] |
| 18:31 |
<andrewbogott> |
depooling tools-exec-1422 and tools-webgrid-lighttpd-1413 for host reboot |
[tools] |
| 18:26 |
<andrewbogott> |
repooling tools-exec-1404 and 1434 for host reboot |
[tools] |
| 18:06 |
<andrewbogott> |
depooling tools-exec-1404 and 1434 for host reboot |
[tools] |
| 18:04 |
<andrewbogott> |
repooling tools-exec-1402, 1426, 1429, 1433, tools-webgrid-lighttpd-1408, 1414, 1424 |
[tools] |
| 17:48 |
<andrewbogott> |
depooling tools-exec-1402, 1426, 1429, 1433, tools-webgrid-lighttpd-1408, 1414, 1424 |
[tools] |
| 17:28 |
<andrewbogott> |
disabling tools-webgrid-generic-1402, tools-webgrid-lighttpd-1403, tools-exec-1403 for host reboot |
[tools] |
| 17:26 |
<andrewbogott> |
repooling tools-exec-1405, 1425, tools-webgrid-generic-1403, tools-webgrid-lighttpd-1401, 1405 after host reboot |
[tools] |
| 17:08 |
<andrewbogott> |
depooling tools-exec-1405, 1425, tools-webgrid-generic-1403, tools-webgrid-lighttpd-1401, 1405 for host reboot |
[tools] |
| 16:19 |
<andrewbogott> |
repooling tools-exec-1401, 1407, 1408, 1430, 1431, 1432, 1435, 1438, 1439, 1441, tools-webgrid-lighttpd-1402, 1407 after host reboot |
[tools] |
| 15:52 |
<andrewbogott> |
depooling tools-exec-1401, 1407, 1408, 1430, 1431, 1432, 1435, 1438, 1439, 1441, tools-webgrid-lighttpd-1402, 1407 for host reboot |
[tools] |
| 13:35 |
<chasemp> |
tools-mail almouked@ltnet.net 719 pending messages cleared |
[tools] |
|
2018-01-10
§
|
| 15:14 |
<chasemp> |
tools-clushmaster-01:~$ clush -f 1 -w @k8s-worker "sudo puppet agent --enable && sudo puppet agent --test" |
[tools] |
| 15:03 |
<chasemp> |
tools-k8s-master-01:~# for n in `kubectl get nodes | awk '{print $1}' | grep -v -e tools-worker-1001 -e tools-worker-1016 -e tools-worker-1016`; do kubectl cordon $n; done |
[tools] |
| 14:41 |
<chasemp> |
tools-clushmaster-01:~$ clush -w @k8s-worker "sudo puppet agent --disable 'chase rollout'" |
[tools] |
| 14:01 |
<chasemp> |
tools-k8s-master-01:~# kubectl uncordon tools-worker-1001.tools.eqiad.wmflabs |
[tools] |
| 13:57 |
<arturo> |
T184604 cleaned stalled log files that prevented logrotate from working. Triggered a couple of logrorate runs by hand in tools-worker-1020.tools.eqiad.wmflabs |
[tools] |
| 13:46 |
<arturo> |
T184604 aborrero@tools-k8s-master-01:~$ sudo kubectl uncordon tools-worker-1020.tools.eqiad.wmflabs |
[tools] |
| 13:45 |
<arturo> |
T184604 aborrero@tools-worker-1020:/var/log$ sudo mkdir /var/lib/kubelet/pods/bcb36fe1-7d3d-11e7-9b1a-fa163edef48a/volumes |
[tools] |
| 13:26 |
<arturo> |
sudo kubectl drain tools-worker-1020.tools.eqiad.wmflabs |
[tools] |
| 13:22 |
<arturo> |
empty by hand syslog and daemon.log files. They are so big that logrotate won't handle them |
[tools] |
| 13:20 |
<arturo> |
aborrero@tools-worker-1020:~$ sudo service kubelet restart |
[tools] |
| 13:18 |
<arturo> |
aborrero@tools-k8s-master-01:~$ sudo kubectl cordon tools-worker-1020.tools.eqiad.wmflabs for T184604 |
[tools] |
| 13:13 |
<arturo> |
detected low space in tools-worker-1020, big files in /var/log due to kubelet issue. Opened T184604 |
[tools] |