151-200 of 943 results (13ms)
2020-02-11 §
23:56 <bd808> Depooling tools-sgewebgrid-lighttpd-0926 (T244791) [tools]
23:38 <bd808> Depooling tools-sgewebgrid-lighttpd-0927 (T244791) [tools]
2020-02-10 §
22:51 <bstorm_> all docker images now use webservice 0.62 [tools]
22:01 <bd808> Manually starting webservices for tools that were running on tools-sgewebgrid-lighttpd-0928 (T244791) [tools]
21:47 <bd808> Depooling tools-sgewebgrid-lighttpd-0928 (T244791) [tools]
2020-02-06 §
05:07 <andrewbogott> cleared out old /tmp and /var/log files on tools-sgebastion-07 [tools]
2020-02-03 §
12:48 <arturo> shutdown tools-prometheus-01 and tools-prometheus-02, after fixing the proxy `tools-prometheus.wmflabs.org` to tools-prometheus-03, data synced (T238096) [tools]
2020-01-30 §
21:04 <andrewbogott> also apt-get install python3-novaclient on tools-prometheus-03 and tools-prometheus-04 to suppress cronspam. Possible real fix for this is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569084/ [tools]
20:39 <andrewbogott> apt-get install python3-keystoneclient on tools-prometheus-03 and tools-prometheus-04 to suppress cronspam. Possible real fix for this is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569084/ [tools]
13:14 <arturo> drop floating IP 185.15.56.60 and FQDN `prometheus.tools.wmcloud.org` because this is not how the prometheus setup is right now. Use a web proxy instead `tools-prometheus-new.wmflabs.org` (T238096) [tools]
12:57 <arturo> created domain `tools.wmcloud.org` in the tools project after some back and forth with designated, permissions and the database. I plan to use this domain to test the new Debian Buster-based prometheus setup (T238096) [tools]
2020-01-29 §
20:07 <bd808> Created {bastion,login,dev}.toolforge.org service names for Toolforge bastions using Horizon & Designate [tools]
2020-01-27 §
07:05 <zhuyifei1999_> wrong package. uninstalled. the correct one is bpfcc-tools and seems only available in buster+. T115231 [tools]
2020-01-24 §
20:58 <bd808> Built tools-k8s-worker-21 to test out build script following openstack client upgrade [tools]
15:45 <bd808> Rebuilding all Docker containers again because I failed to actually update the build server git clone properly last time I did this [tools]
05:23 <bd808> Building 6 new tools-k8s-worker instances for the 2020 Kubernetes cluster (take 2) [tools]
04:41 <bd808> Rebuilding all Docker images to pick up webservice-python-bootstrap changes [tools]
2020-01-23 §
23:38 <bd808> Halted tools-k8s-worker build script after first instance (tools-k8s-worker-10) stuck in "scheduling" state for 20 minutes [tools]
23:16 <bd808> Building 6 new tools-k8s-worker instances for the 2020 Kubernetes cluster [tools]
05:15 <bd808> Building tools-elastic-04 [tools]
04:39 <bd808> wmcs-openstack quota set --instances 192 [tools]
04:36 <bd808> wmcs-openstack quota set --cores 768 --ram 1536000 [tools]
2020-01-21 §
19:17 <bstorm_> depooled and rebooted tools-sgeexec-0914 because it was acting funny [tools]
18:30 <bstorm_> depooling and rebooting tools-sgeexec-[0911,0913,0919,0921,0924,0931,0933,0935,0939,0941].tools.eqiad.wmflabs [tools]
2020-01-13 §
17:48 <bd808> Running `puppet ca destroy` for each unsigned cert on tools-puppetmaster-01 (T242642) [tools]
16:42 <bd808> Cordoned and fixed puppet on tools-k8s-worker-12. Rebooting now. T242559 [tools]
16:33 <bd808> Cordoned and fixed puppet on tools-k8s-worker-11. Rebooting now. T242559 [tools]
16:31 <bd808> Cordoned and fixed puppet on tools-k8s-worker-10. Rebooting now. T242559 [tools]
16:26 <bd808> Cordoned and fixed puppet on tools-k8s-worker-9. Rebooting now. T242559 [tools]
2020-01-12 §
22:31 <Krenair> same on -13 and -14 [tools]
2020-01-11 §
01:33 <bstorm_> updated toollabs-webservice package to 0.57, which should allow persisting mem and cpu in manifests with burstable qos. [tools]
2020-01-10 §
15:35 <bstorm_> depooling and rebooting tools-worker-1016 because it still had the leftover mount problems [tools]
15:30 <bstorm_> git stash-ing local puppet changes in hopes that arturo has that material locally, and it doesn't break anything to do so [tools]
2020-01-09 §
23:35 <bstorm_> depooled tools-sgeexec-0939 because it isn't acting right and rebooting it [tools]
2020-01-07 §
22:40 <bstorm_> rebooted tools-worker-1007 to recover it from disk full and general badness [tools]
15:46 <bd808> Rebooting tools-k8s-worker-[6-14] [tools]
13:23 <arturo> [new k8s] doing changes to kube-state-metrics and metrics-server trying to relocate them to the 'metrics' namespace (T241853) [tools]
05:28 <bd808> Creating tools-k8s-worker-[6-14] (again) [tools]
05:20 <bd808> Deleting busted tools-k8s-worker-[6-14] [tools]
05:02 <bd808> Creating tools-k8s-worker-[6-14] [tools]
00:15 <bstorm_> moving tools-sgewebgrid-lighttpd-0918 and -0919 to cloudvirt1004 from cloudvirt1029 to rebalance load [tools]
00:02 <bstorm_> depooled tools-sgewebgrid-lighttpd-0918 and 0919 to move to cloudvirt1004 to improve spread [tools]
2020-01-06 §
23:40 <bd808> Deleted tools-sgewebgrid-lighttpd-09{0[1-9],10} [tools]
23:36 <bd808> Shutdown tools-sgewebgrid-lighttpd-09{0[1-9],10} [tools]
23:34 <bd808> Decommissioned tools-sgewebgrid-lighttpd-09{0[1-9],10} [tools]
23:01 <bd808> Depooled tools-sgewebgrid-lighttpd-0910.tools.eqiad.wmflabs [tools]
22:58 <bd808> Depooling tools-sgewebgrid-lighttpd-090[2-9] [tools]
22:57 <bd808> Disabling queues on tools-sgewebgrid-lighttpd-090[2-9] [tools]
21:07 <bd808> Restarted kube2proxy on tools-proxy-05 to try and refresh admin tool's routes [tools]
18:54 <bstorm_> edited /etc/fstab to remove NFS and unmounted the nfs volumes tools-k8s-haproxy-1 T241908 [tools]