|
2021-04-06
§
|
| 11:45 |
<arturo> |
upgrading jobutils & misctools to 1.42 everywhere |
[tools] |
| 11:39 |
<arturo> |
cleaning up aptly: old package versions, old repos (jessie, trusty, precise) etc |
[tools] |
| 10:31 |
<dcaro> |
Removing etcd member tools-k8s-etcd-6.tools.eqiad.wmflabs (T267082) |
[tools] |
| 10:21 |
<arturo> |
published jobutils & misctools 1.42 (T278748) |
[tools] |
| 10:21 |
<arturo> |
published jobutils & misctools 1.42 |
[tools] |
| 10:21 |
<arturo> |
aptly repo had some weirdness due to the cinder volume: hardlinks created by aptly were broken, solved with `sudo aptly publish --skip-signing repo stretch-tools -force-overwrite` |
[tools] |
| 10:07 |
<dcaro> |
adding new etcd member using the cookbook wmcs.toolforge.add_etcd_node (T267082) |
[tools] |
| 10:05 |
<arturo> |
installed aptly from buster-backports on tools-services-05 to see if that makes any difference with an issue when publishing repos |
[tools] |
| 09:53 |
<dcaro> |
Removing etcd member tools-k8s-etcd-4.tools.eqiad.wmflabs (T267082) |
[tools] |
| 08:55 |
<dcaro> |
adding new etcd member using the cookbook wmcs.toolforge.add_etcd_node (T267082) |
[tools] |
|
2021-03-31
§
|
| 15:57 |
<arturo> |
rebooting `tools-mail-03` after enabling NFS (T267082, T278538) |
[tools] |
| 15:57 |
<arturo> |
rebooting `tools-mail-03` after enabling NFS (T |
[tools] |
| 15:04 |
<arturo> |
created MX record for `tools.wmcloud.org` pointing to `mail.tools.wmcloud.org` |
[tools] |
| 15:03 |
<arturo> |
created DNS A record `mail.tools.wmcloud.org` pointing to 185.15.56.63 |
[tools] |
| 14:56 |
<arturo> |
shutoff tools-mail-02 (T278538) |
[tools] |
| 14:54 |
<arturo> |
point floating IP 185.15.56.63 to tools-mail-03 (T278538) |
[tools] |
| 14:45 |
<arturo> |
created VM `tools-mail-03` as Debian Buster (T278538) |
[tools] |
| 14:39 |
<arturo> |
relocate some of the hiera keys for email server from project-level to prefix |
[tools] |
| 09:44 |
<dcaro> |
running disk performance test on etcd-4 (round2) |
[tools] |
| 09:05 |
<dcaro> |
running disk performance test on etcd-8 |
[tools] |
| 08:43 |
<dcaro> |
running disk performance test on etcd-4 |
[tools] |
|
2021-03-30
§
|
| 16:15 |
<bstorm> |
added `labstore::traffic_shaping::egress: 800mbps` to tools-static prefix T278539 |
[tools] |
| 15:44 |
<arturo> |
shutoff tools-static-12/13 (T278539) |
[tools] |
| 15:41 |
<arturo> |
point horizon web proxy `tools-static.wmflabs.org` to tools-static-14 (T278539) |
[tools] |
| 15:37 |
<arturo> |
add `mount_nfs: true` to tools-static prefix (T2778539) |
[tools] |
| 15:26 |
<arturo> |
create VM tools-static-14 with Debian Buster image (T278539) |
[tools] |
| 12:19 |
<arturo> |
introduce horizon proxy `deb-tools.wmcloud.org` (T278436) |
[tools] |
| 12:15 |
<arturo> |
shutdown tools-sgebastion-09 (stretch) |
[tools] |
| 11:05 |
<arturo> |
created VM `tools-sgebastion-10` as Debian Buster (T275865) |
[tools] |
| 11:04 |
<arturo> |
created server group `tools-bastion` with anti-affinity policy |
[tools] |
|
2021-03-25
§
|
| 19:30 |
<bstorm> |
forced deletion of all jobs stuck in a deleting state T277653 |
[tools] |
| 17:46 |
<arturo> |
rebooting tools-sgeexec-* nodes to account for new grid master (T277653) |
[tools] |
| 16:20 |
<arturo> |
rebuilding tools-sgegrid-master VM as debian buster (T277653) |
[tools] |
| 16:18 |
<arturo> |
icinga-downtime toolschecker for 2h |
[tools] |
| 16:05 |
<bstorm> |
failed over the tools grid to the shadow master T277653 |
[tools] |
| 13:36 |
<arturo> |
shutdown tools-sge-services-03 (T278354) |
[tools] |
| 13:33 |
<arturo> |
shutdown tools-sge-services-04 (T278354) |
[tools] |
| 13:31 |
<arturo> |
point aptly clients to `tools-services-05.tools.eqiad1.wikimedia.cloud` (hiera change) (T278354) |
[tools] |
| 12:58 |
<arturo> |
created VM `tools-services-05` as Debian Buster (T278354) |
[tools] |
| 12:51 |
<arturo> |
create cinder volume `tools-aptly-data` (T278354) |
[tools] |