851-900 of 3116 results (20ms)
2019-12-12 §
22:53 <bstorm_> rebooting the cron server, tools-sgecron-01 as it wasn't recovered from last night's maintenance [tools]
11:20 <arturo> rolling reboot for all grid & k8s worker nodes due to NFS staleness [tools]
09:22 <arturo> reboot tools-sgeexec-0911 to try fixing weird NFS state [tools]
08:46 <arturo> doing `run-puppet-agent` in all VMs to see state of NFS [tools]
08:34 <arturo> reboot tools-worker-1033/1034 and tools-sgebastion-08 to try to correct NFS mount issues [tools]
2019-12-11 §
18:13 <bd808> Restarted maintain-dbusers on labstore1004. Process had not logged any account creations since 2019-12-01T22:45:45. [tools]
17:24 <andrewbogott> deleted and/or truncated a bunch of logfiles on tools-worker-1031 [tools]
2019-12-10 §
13:59 <arturo> set pod replicas to 3 in the new k8s cluster (T239405) [tools]
2019-12-09 §
11:06 <andrewbogott> deleting unused security groups: catgraph, devpi, MTA, mysql, syslog, test T91619 [tools]
2019-12-04 §
13:45 <arturo> drop puppet prefix `tools-cron`, deprecated and no longer in use [tools]
2019-11-29 §
11:45 <arturo> created 3 new VMs `tools-k8s-worker-[3,4,5]` (T239403) [tools]
10:28 <arturo> re-arm keyholder in tools-acme-chief-01 (password in labs/private.git @ tools-puppetmaster-01) [tools]
10:27 <arturo> re-arm keyholder in tools-acme-chief-02 (password in labs/private.git @ tools-puppetmaster-01) [tools]
2019-11-26 §
23:25 <bstorm_> rebuilding docker images to include the new webservice 0.52 in all versions instead of just the stretch ones T236202 [tools]
22:57 <bstorm_> push upgraded webservice 0.52 to the buster and jessie repos for container rebuilds T236202 [tools]
19:55 <phamhi> drained tools-worker-1002,8,15,32 to rebalance the cluster [tools]
19:45 <phamhi> cleaned up container that was taken up 16G of disk space on tools-worker-1020 in order to re-run puppet client [tools]
14:01 <arturo> drop hiera references to `tools-test-proxy-01.tools.eqiad.wmflabs`. Such VM no longer exists [tools]
14:00 <arturo> introduce the `profile::toolforge::proxies` hiera key in the global puppet config [tools]
2019-11-25 §
10:35 <arturo> refresh puppet certs for tools-k8s-etcd-[4-6] nodes (T238655) [tools]
10:35 <arturo> add puppet cert SANs via instance hiera to tools-k8s-etcd-[4-6] nodes (T238655) [tools]
2019-11-22 §
13:32 <arturo> created security group `tools-new-k8s-full-connectivity` and add new k8s VMs to it (T238654) [tools]
05:55 <jeh> add Riley Huntley `riley` to base tools project [tools]
2019-11-21 §
12:48 <arturo> reboot the new k8s cluster after the upgrade [tools]
11:49 <arturo> upgrading new k8s kubectl version to 1.15.6 (T238654) [tools]
11:44 <arturo> upgrading new k8s kubelet version to 1.15.6 (T238654) [tools]
10:29 <arturo> upgrading new k8s cluster version to 1.15.6 using kubeadm (T238654) [tools]
10:28 <arturo> install kubeadm 1.15.6 on worker/control nodes in the new k8s cluster (T238654) [tools]
2019-11-19 §
13:49 <arturo> re-create nginx-ingress pod due to deployment template refresh (T237643) [tools]
12:46 <arturo> deploy changes to tools-prometheus to account for the new k8s cluster (T237643) [tools]
2019-11-15 §
14:44 <arturo> stop live-hacks on tools-prometheus-01 T237643 [tools]
2019-11-13 §
17:20 <arturo> live-hacking tools-prometheus-01 to test some experimental configs for the new k8s cluster (T237643) [tools]
2019-11-12 §
12:52 <arturo> reboot tools-proxy-06 to reset iptables setup T238058 [tools]
2019-11-10 §
02:17 <bd808> Building new Docker images for T237836 (retrying after cleaning out old images on tools-docker-builder-06) [tools]
02:15 <bd808> Cleaned up old images on tools-docker-builder-06 using instructions from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Building_toolforge_specific_images [tools]
02:10 <bd808> Building new Docker images for T237836 [tools]
01:45 <bstorm_> deploying bugfix for webservice in tools and toolsbeta T237836 [tools]
2019-11-08 §
22:47 <bstorm_> adding rsync::server::wrap_with_stunnel: false to the tools-docker-registry-03/4 servers to unbreak puppet [tools]
18:40 <bstorm_> pushed new webservice package to the bastions T230961 [tools]
18:37 <bstorm_> pushed new webservice package supporting buster containers to repo T230961 [tools]
18:36 <bstorm_> pushed buster-sssd images to the docker repo [tools]
17:15 <phamhi> pushed new buster images with the prefix name "toolforge" [tools]
2019-11-07 §
13:27 <arturo> deployed registry-admission-webhook and ingress-admission-controller into the new k8s cluster (T236826) [tools]
13:01 <arturo> creating puppet prefix `tools-k8s-worker` and a couple of VMs `tools-k8s-worker-[1,2]` T236826 [tools]
12:57 <arturo> increasing project quota T237633 [tools]
11:54 <arturo> point `k8s.tools.eqiad1.wikimedia.cloud` to tools-k8s-haproxy-1 T236826 [tools]
11:43 <arturo> create VMs `tools-k8s-haproxy-[1,2]` T236826 [tools]
11:43 <arturo> create puppet prefix `tools-k8s-haproxy` T236826 [tools]
2019-11-06 §
22:32 <bstorm_> added rsync::server::wrap_with_stunnel: false to tools-sge-services prefix to fix puppet [tools]
21:33 <bstorm_> docker images needed for kubernetes cluster upgrade deployed T215531 [tools]