|
2019-03-14
§
|
| 23:52 |
<bd808> |
Disabled job queues and rescheduled continuous jobs away from tools-exec-14{21,22,23,24,25,26,27,28,29,30,31,32} (T217152) |
[tools] |
| 23:28 |
<bd808> |
Deleted tools-bastion-05 (T217152) |
[tools] |
| 22:30 |
<bd808> |
Removed obsolete submit hosts from Trusty grid config |
[tools] |
| 22:20 |
<bd808> |
Removed tools-webgrid-lighttpd-142{0,1,2,5} from the grid and shutdown instances via horizon (T217152) |
[tools] |
| 22:10 |
<bd808> |
Depooled tools-webgrid-lighttpd-142{0,1,2,5} (T217152) |
[tools] |
| 21:55 |
<bd808> |
Removed submit host flag from tools-bastion-05.tools.eqiad.wmflabs, removed floating ip, and shutdown instance via horizon (T217152) |
[tools] |
| 21:48 |
<bd808> |
Removed tools-exec-14{33,34,35,36,37,38,39,40,41,42} from the grid and shutdown instances via horizon (T217152) |
[tools] |
| 21:38 |
<gtirloni> |
rebooted tools-sgewebgrid-generic-0904 (T218341) |
[tools] |
| 21:32 |
<gtirloni> |
rebooted tools-exec-1020 (T218341) |
[tools] |
| 21:23 |
<gtirloni> |
rebooted tools-sgeexec-0919, tools-sgeexec-0934, tools-worker-1018 (T218341) |
[tools] |
| 21:19 |
<bd808> |
Killed jobs still running on tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs 2 weeks after being depooled (T217152) |
[tools] |
| 20:58 |
<bd808> |
Repooled tools-sgeexec-0941 following reboot |
[tools] |
| 20:57 |
<bd808> |
Hard reboot of tools-sgeexec-0941 via horizon |
[tools] |
| 20:54 |
<bd808> |
Depooled and rebooted tools-sgeexec-0941.tools.eqiad.wmflabs |
[tools] |
| 20:53 |
<bd808> |
Repooled tools-sgeexec-0917 following reboot |
[tools] |
| 20:52 |
<bd808> |
Hard reboot of tools-sgeexec-0917 via horizon |
[tools] |
| 20:47 |
<bd808> |
Depooled and rebooted tools-sgeexec-0917 |
[tools] |
| 20:44 |
<bd808> |
Repooled tools-sgeexec-0908 after reboot |
[tools] |
| 20:36 |
<bd808> |
depooled and rebooted tools-sgeexec-0908 |
[tools] |
| 19:08 |
<gtirloni> |
rebooted tools-worker-1028 (T218341) |
[tools] |
| 19:08 |
<gtirloni> |
rebooted tools-sgewebgrid-lighttpd-0914 (T218341) |
[tools] |
| 19:07 |
<gtirloni> |
rebooted tools-sgewebgrid-lighttpd-0914 |
[tools] |
| 18:13 |
<gtirloni> |
drained tools-worker-1028 for reboot (processes in D state) |
[tools] |
|
2019-03-13
§
|
| 23:30 |
<bd808> |
Rebuilding stretch Kubernetes images |
[tools] |
| 22:55 |
<bd808> |
Rebuilding jessie Kubernetes images |
[tools] |
| 17:11 |
<bstorm_> |
specifically rebooted SGE cron server tools-sgecron-01 |
[tools] |
| 17:10 |
<bstorm_> |
rebooted cron server |
[tools] |
| 16:10 |
<bd808> |
Updated DNS for dev.tools.wmflabs.org to point to Stretch secondary bastion. This was missed on 2019-03-07 |
[tools] |
| 12:33 |
<arturo> |
reboot tools-sgebastion-08 (T215154) |
[tools] |
| 12:17 |
<arturo> |
reboot tools-sgebastion-07 (T215154) |
[tools] |
| 11:53 |
<arturo> |
enable puppet in tools-sgebastion-07 (T215154) |
[tools] |
| 11:20 |
<arturo> |
disable puppet in tools-sgebastion-07 for testing T215154 |
[tools] |
| 05:07 |
<bstorm_> |
re-enabled puppet for tools-sgebastion-07 |
[tools] |
| 04:59 |
<bstorm_> |
disabled puppet for a little bit on tools-bastion-07 |
[tools] |
| 00:22 |
<bd808> |
Raise web-memlimit for isbn tool to 6G for tomcat8 (T217406) |
[tools] |