2018-03-12
§
|
20:09 |
<madhuvishy> |
Run clush -w @all -b 'sudo umount /mnt/nfs/labstore1003-scratch && sudo mount -a' to remount scratch across all of tools |
[tools] |
17:13 |
<arturo> |
T188994 upgrading packages from `stable` |
[tools] |
16:53 |
<arturo> |
T188994 upgrading packages from stretch-wikimedia |
[tools] |
16:33 |
<arturo> |
T188994 upgrading packages form jessie-wikimedia |
[tools] |
14:58 |
<zhuyifei1999_> |
building, publishing, and deploying misctools 1.31 5f3561e T189430 |
[tools] |
13:31 |
<arturo> |
tools-exec-1441 and tools-exec-1442 rebooted fine and are repooled |
[tools] |
13:26 |
<arturo> |
depool tools-exec-1441 and tools-exec-1442 for reboots |
[tools] |
13:19 |
<arturo> |
T188994 upgrade packages from jessie-backports in all jessie servers |
[tools] |
12:49 |
<arturo> |
T188994 upgrade packages from trusty-updates in all ubuntu servers |
[tools] |
12:34 |
<arturo> |
T188994 upgrade packages from trusty-wikimedia in all ubuntu servers |
[tools] |
2018-03-06
§
|
16:15 |
<madhuvishy> |
Reboot tools-docker-registry-02 T189018 |
[tools] |
15:50 |
<madhuvishy> |
Rebooting tools-worker-1011 |
[tools] |
15:08 |
<chasemp> |
tools-k8s-master-01:~# kubectl uncordon tools-worker-1011.tools.eqiad.wmflabs |
[tools] |
15:03 |
<arturo> |
drain and reboot tools-worker-1011 |
[tools] |
15:03 |
<chasemp> |
rebooted tools-worker 1001-1008 |
[tools] |
14:58 |
<arturo> |
drain and reboot tools-worker-1010 |
[tools] |
14:27 |
<chasemp> |
multiple tools running on k8s workers report issues reading replica.my.cnf file atm |
[tools] |
14:27 |
<chasemp> |
reboot tools-worker-100[12] |
[tools] |
14:23 |
<chasemp> |
downtime icinga alert for k8s workers ready |
[tools] |
13:21 |
<arturo> |
T188994 in some servers there was some race in the dpkg lock between apt-upgrade and puppet. Also, I forgot to use DEBIAN_FRONTEND=noninteractive, so debconf prompts happened and stalled dpkg operations. Already solved, but some puppet alerts were produced |
[tools] |
12:58 |
<arturo> |
T188994 upgrading packages in jessie nodes from the oldstable source |
[tools] |
11:42 |
<arturo> |
clush -w @all "sudo DEBIAN_FRONTEND=noninteractive apt-get autoclean" <-- free space in filesystem |
[tools] |
11:41 |
<arturo> |
aborrero@tools-clushmaster-01:~$ clush -w @all "sudo DEBIAN_FRONTEND=noninteractive apt-get autoremove -y" <-- we did in canary servers last week and it went fine. So run in fleet-wide |
[tools] |
11:36 |
<arturo> |
(ubuntu) removed linux-image-3.13.0-142-generic and linux-image-3.13.0-137-generic (T188911) |
[tools] |
11:33 |
<arturo> |
removing unused kernel packages in ubuntu nodes |
[tools] |