151-200 of 290 results (12ms)
2020-11-10 §
16:41 <arturo> set paws in sqlite mode because T266587 (kubectl --namespace prod edit configmap hub-config) [paws]
2020-10-15 §
19:12 <andrewbogott> uncordoned paws-k8s-worker-1 and -2 [paws]
18:48 <andrewbogott> draining paws-k8s-worker-2 for move to ceph [paws]
18:36 <andrewbogott> draining paws-k8s-worker-1 for move to ceph [paws]
2020-09-29 §
10:59 <arturo> last 2 commands should help puppet agent in the paws project, previously it had issues fetching acme-chief certs because an API update [paws]
10:58 <arturo> aborrero@paws-acme-chief-01:~$ sudo systemctl restart uwsgi-acme-chief.service [paws]
10:56 <arturo> aborrero@paws-acme-chief-01:~$ sudo systemctl restart acme-chief.service [paws]
2020-08-14 §
17:09 <bstorm> backing up the old proxy config to NFS and deleting paws-proxy-02 T211096 [paws]
2020-08-07 §
22:30 <bstorm> removing downtime for paws and front page monitor T211096 [paws]
18:01 <bstorm> shutting down paws-proxy-02 T211096 [paws]
17:05 <bstorm> running the final rsync to the new cluster's nfs T211096 [paws]
16:08 <bstorm> changing paws.wmflabs.org to point at the new cluster ip 185.15.56.57 T211096 [paws]
16:02 <bstorm> LAST MESSAGE WRONG: switching NEW cluster to toolsdb T211096 [paws]
16:02 <bstorm> switching old cluster to toolsdb T211096 [paws]
15:58 <bstorm> switching old cluster to sqlite T211096 [paws]
15:53 <bstorm> downtiming alerts in case they need changes (seems likely) T211096 [paws]
2020-07-30 §
20:40 <bstorm> upgrading the singleuser image to test shuffling around some of the pip installs [paws]
16:38 <bstorm> removing the *.paws.wmflabs.org SNI name because it won't be used and it might trigger a re-issue of certs T255249 [paws]
15:39 <bstorm> upgrading acme-chief to 0.27-1 [paws]
2020-07-29 §
18:03 <bstorm> powering on paws-k8s-haproxy-1 because that worked fine [paws]
18:00 <bstorm> powering off paws-k8s-haproxy-1 to test failover [paws]
2020-07-24 §
17:25 <bstorm> to force repulling of every image everywhere, uninstalling paws in the new cluster and reinstalling it T258812 [paws]
09:39 <arturo> dropped the DNS wildcard record `*.paws.wmcloud.org IN A 185.15.56.57` and created concrete CNAME records for the FQDNs we actually use (T211096) [paws]
2020-07-23 §
22:51 <bstorm> deploying via the default 'latest' tag in the new cluster T211096 [paws]
22:48 <bstorm> tagged the newbuild tags with "latest" to set sane defaults for all images in the helm chart T211096 [paws]
21:14 <bstorm> pushing quay.io/wikimedia-paws-prod/nbserve:newbuild to main repo T211096 [paws]
21:11 <bstorm> pushing quay.io/wikimedia-paws-prod/deploy-hook:newbuild to main repo T211096 [paws]
21:09 <bstorm> pushing quay.io/wikimedia-paws-prod/singleuser:newbuild to the main repo T211096 [paws]
21:08 <bstorm> pushing quay.io/wikimedia-paws-prod/paws-hub:newbuild to the main repo T211096 [paws]
21:06 <bstorm> pushing dbproxy docker image for new cluster into main quay.io repo T211096 [paws]
2020-07-22 §
23:32 <bstorm> setting the default NFS version to 4.2 while excepting the two stretch servers T257945 [paws]
2020-07-21 §
15:13 <chicocvenancio> merge pr #50 to fix T258142 [paws]
2020-07-06 §
21:41 <bstorm> deployed ingress to redirect paws.wmcloud.org to the wikitech doc page T195217 [paws]
2020-06-30 §
23:00 <bstorm> added paws-public.wmflabs.org to the alt-names for acme-chief, which broke it until we hand off the zone to the paws project <sorry!> T195217 T255997 [paws]
2020-06-26 §
21:57 <bstorm> applied the metrics manifests to kubernetes to enable metrics-server, cadvisor, etc. T256361 [paws]
2020-06-25 §
22:52 <bstorm> created paws-k8s-worker-5/6/7 as x-large nodes to bring the cluster up to roughly the same capacity as the existing one using soft anti-affinity T211096 T253267 [paws]
22:43 <bstorm> bumped quota up to 24 instances, 128 GB RAM and 56 cores T211096 [paws]
16:39 <bstorm> deleted the deployhook from the in-progress new cluster for now just in case T211096 [paws]
15:44 <bstorm> deployed a proof-of-concept paws-public setup in the new cluster T255997 [paws]
2020-06-24 §
23:18 <bstorm> added A record for *.paws.wmcloud.org to public and hub to use T211096 T255997 T195217 [paws]
21:45 <bstorm> doing an initial rsync of the paws userhomes to the new project T160113 [paws]
2020-06-19 §
10:00 <arturo> enabled `paws.wmflabs.org` and `*.paws.wmflabs.org` as valid ingress domains (acme-chief TLS cert, haproxy, etc) (T195217) [paws]
2020-06-17 §
21:51 <bstorm_> upgraded chart in the new cluster to include resource limits T251298 [paws]
21:51 <bstorm_> upgraded chart in the new cluster to include resource limits [paws]
2020-06-16 §
15:48 <arturo> change DNS record k8s.svc.paws.eqiad1.wikimedia.cloud to point to the haproxy VIP port address 172.16.1.171 (T195217) [paws]
15:47 <arturo> associate floating IP 185.15.56.57 with haproxy VIP port (T295217) [paws]
15:43 <arturo> allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 1b40be58-7182-41aa-95ce-797f94f83d66` (T295217) [paws]
15:43 <arturo> allow traffic to haproxy VM ports from the VIP port: `sudo wmcs-openstack port set --allowed-address ip-address=172.16.1.171 9ccc43d9-1a8a-4287-afda-67e8bab27a9f` (T295217) [paws]
15:37 <arturo> `aborrero@cloudcontrol1004:~ 1 $ sudo wmcs-openstack --os-project-id=paws port create --network 7425e328-560c-4f00-8e99-706f3fb90bb4 paws-haproxy-vip` (T295217) [paws]
15:23 <arturo> live-hacking paws-puppetmaster-01 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/605944 for T195217 [paws]