1-50 of 176 results (22ms)
2021-06-30 §
20:05 <bstorm> tried force delete on the ingress-nginx-gen2 namespace, which doesn't appear to be working either until metrics-server is fixed T285905 [paws]
20:00 <bstorm> renewed k8s metrics-server certs and the deployment [paws]
18:04 <majavah> renew kubernetes metrics-server certificate [paws]
17:26 <majavah> creating paws-k8s-ingress-[3-4] and joining them to the k8s cluster T264221 [paws]
17:16 <bstorm> temporarily increased quota to 60 cores to enable T264221 [paws]
2021-06-03 §
20:43 <chicocvenancio> tagged new singleuser image, fixes T283969 [paws]
2021-05-27 §
21:53 <bstorm> added paws-k8s-control-2.paws.eqiad.wmflabs back to the list of control nodes at the proxy [paws]
21:50 <bstorm> renewed the certs for paws-k8s-control-2 [paws]
20:37 <bstorm> removed paws-k8s-control-2.paws.eqiad.wmflabs from the proxy because it is somewhat broken (certs expired) [paws]
19:41 <bstorm> forced removal of openrefine in paws for now and deleted all current user server pods to force use of the new image [paws]
2021-05-23 §
01:45 <bstorm> manually pulling updated singleuser images because I hate waiting to test things [paws]
01:36 <bstorm> update latest tag for singleuser image to b894471 for https://github.com/toolforge/paws/pull/69 [paws]
00:56 <bstorm> deploying changes for https://github.com/toolforge/paws/pull/67 [paws]
2021-05-21 §
00:06 <bstorm> creating trove mysql instance pawsdb-1 T267683 [paws]
2021-05-12 §
19:33 <bstorm> added taavi to paws.admin [paws]
2021-05-11 §
09:17 <Majavah> set `profile::wmcs::kubeadm::docker_vol: false` on ingress nodes T282087 [paws]
09:15 <arturo> added user `taavi` (Majavah) as projectadmin [paws]
2021-04-20 §
23:18 <bstorm> deploying updated paws-hub image for https://github.com/toolforge/paws/pull/63 [paws]
2021-04-02 §
21:50 <bstorm> deploying latest PRs to add a note on the wikireplicas changes [paws]
2020-12-21 §
20:27 <bstorm> applied tuning for timeouts and elections on the k8s etcd pods of 300 for heartbeat and 3000 for elections T267966 [paws]
2020-12-17 §
02:22 <bstorm> Set PAWS hub back to using mariadb T266587 [paws]
2020-12-16 §
18:21 <chicocvenancio> move paws to sqlite while toolsdb is down. [paws]
2020-12-10 §
17:00 <arturo> fixing /etc/kubernetes/kublet.conf and restarting kubelet in paws-k8s-control-1 (T269865) [paws]
2020-12-05 §
00:42 <bd808> `kubectl delete po renderer-794886b9cd-9nc6c -n prod` after seeing lots of listen queue full errors in the pod logs. [paws]
2020-11-30 §
18:22 <bstorm> 1.17 upgrade for kubernetes complete T268669 [paws]
17:25 <bstorm> upgrading the worker nodes (this will likely kill services briefly when some pods are rescheduled) T268669 [paws]
17:14 <bstorm> updated the calico-kube-controllers deployment to use our internal registry to deal with docker-hub rate-limiting T268669 T269016 [paws]
17:08 <chicocvenancio> delete orphaned jupyter server pod `kubectl -n prod delete pod jupyter--45volutionoftheuniverse`. Respective server not running in jupyter admin UI. [paws]
16:31 <bstorm> upgrading pods on paws-k8s-control-3 T268669 [paws]
16:17 <bstorm> starting upgrade on paws-k8s-control-2 T268669 (first kubectl drain paws-k8s-control-2 --ignore-daemonsets) [paws]
15:53 <bstorm> proceeding with upgrade to 1.17 on paws-k8s-control-1 T268669 [paws]
15:49 <bstorm> draining paws-k8s-control-1 for upgrade T268669 [paws]
12:49 <arturo> disable puppet in all k8s nodes to prepare for the upgrade (T268669) [paws]
12:49 <arturo> set hiera `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-17'` at project level (T268669) [paws]
2020-11-16 §
22:13 <bstorm> deploying new paws changes for multiinstance readiness [paws]
2020-11-10 §
20:16 <chicocvenancio> restart hub to apply move to sqlite. T267667 [paws]
16:41 <arturo> set paws in sqlite mode because T266587 (kubectl --namespace prod edit configmap hub-config) [paws]
2020-10-15 §
19:12 <andrewbogott> uncordoned paws-k8s-worker-1 and -2 [paws]
18:48 <andrewbogott> draining paws-k8s-worker-2 for move to ceph [paws]
18:36 <andrewbogott> draining paws-k8s-worker-1 for move to ceph [paws]
2020-09-29 §
10:59 <arturo> last 2 commands should help puppet agent in the paws project, previously it had issues fetching acme-chief certs because an API update [paws]
10:58 <arturo> aborrero@paws-acme-chief-01:~$ sudo systemctl restart uwsgi-acme-chief.service [paws]
10:56 <arturo> aborrero@paws-acme-chief-01:~$ sudo systemctl restart acme-chief.service [paws]
2020-08-14 §
17:09 <bstorm> backing up the old proxy config to NFS and deleting paws-proxy-02 T211096 [paws]
2020-08-07 §
22:30 <bstorm> removing downtime for paws and front page monitor T211096 [paws]
18:01 <bstorm> shutting down paws-proxy-02 T211096 [paws]
17:05 <bstorm> running the final rsync to the new cluster's nfs T211096 [paws]
16:08 <bstorm> changing paws.wmflabs.org to point at the new cluster ip 185.15.56.57 T211096 [paws]
16:02 <bstorm> LAST MESSAGE WRONG: switching NEW cluster to toolsdb T211096 [paws]
16:02 <bstorm> switching old cluster to toolsdb T211096 [paws]