2019-02-27
§
|
16:20 |
<zhuyifei1999_> |
regenerating k8s creds for tools.whichsub & tools.permission-denied-test T176027 |
[tools] |
15:40 |
<andrewbogott> |
moving tools-worker-1002, 1005, 1028 to eqiad1-r |
[tools] |
01:35 |
<bd808> |
Shutdown tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs via horizon (T217152) |
[tools] |
01:29 |
<bd808> |
Depooled tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs (T217152) |
[tools] |
01:26 |
<bd808> |
Disabled job queues and rescheduled continuous jobs away from tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs (T217152) |
[tools] |
2019-02-25
§
|
23:20 |
<bstorm_> |
Depooled tools-sgeexec-0914 and tools-sgeexec-0915 for T217066 |
[tools] |
21:41 |
<andrewbogott> |
depooling tools-sgeexec-0911, tools-sgeexec-0912, tools-sgeexec-0913 to test T217066 |
[tools] |
13:11 |
<chicocvenancio> |
PAWS: Stopped AABot notebook pod T217010 |
[tools] |
12:54 |
<chicocvenancio> |
PAWS: Restarted Criscod notebook pod T217010 |
[tools] |
12:21 |
<chicocvenancio> |
PAWS: killed proxy and hub pods to attempt to get it to see routes to open notebooks servers to no avail. Restarted BernhardHumm's notebook pod T217010 |
[tools] |
09:50 |
<gtirloni> |
rebooted tools-sgeexec-09{16,22,40} (T216988) |
[tools] |
09:41 |
<gtirloni> |
rebooted tools-sgeexec-09{16,22,40} |
[tools] |
08:37 |
<zhuyifei1999_> |
uncordon tools-worker-1015.tools.eqiad.wmflabs |
[tools] |
08:34 |
<legoktm> |
hard rebooted tools-worker-1015 via horizon |
[tools] |
07:48 |
<zhuyifei1999_> |
systemd stuck in D state. :( |
[tools] |
07:44 |
<zhuyifei1999_> |
I saved dmesg and process list to a few files in /root if that helps debugging |
[tools] |
07:43 |
<zhuyifei1999_> |
D states are not responding to SIGKILL. Will reboot. |
[tools] |
07:37 |
<zhuyifei1999_> |
tools-worker-1015.tools.eqiad.wmflabs having severe NFS issues (all NFS accessing processes are stuck in D state). Draining. |
[tools] |
2019-02-20
§
|
23:30 |
<zhuyifei1999_> |
begin rebuilding all docker images T178601 T193646 T215683 |
[tools] |
23:25 |
<zhuyifei1999_> |
upgraded toollabs-webservice on tools-bastion-02 to 0.44 (newly-built version) |
[tools] |
23:19 |
<zhuyifei1999_> |
this was built for stretch. hopefully it works for all distros |
[tools] |
23:17 |
<zhuyifei1999_> |
begin build new tools-webservice package T178601 T193646 T215683 |
[tools] |
21:57 |
<andrewbogott> |
moving tools-static-13 to a new virt host |
[tools] |
21:34 |
<andrewbogott> |
moving the tools-static IP from tools-static-13 to tools-static-12 |
[tools] |
19:17 |
<andrewbogott> |
moving tools-bastion-02 to labvirt1004 |
[tools] |
16:56 |
<andrewbogott> |
moving tools-paws-worker-1003 |
[tools] |
15:53 |
<andrewbogott> |
moving tools-worker-1017, tools-worker-1027, tools-worker-1028 |
[tools] |
15:03 |
<andrewbogott> |
moving tools-exec-1413 and tools-exec-1442 |
[tools] |