1-50 of 2045 results (15ms)
2019-09-05 §
21:02 <bd808> Enabled Puppet on tools-docker-registry-03 and forced puppet run (T232135) [tools]
18:13 <bd808> Disabled Puppet on tools-docker-registry-03 to investigate docker-registry issue (no phab task yet) [tools]
2019-09-01 §
20:51 <Reedy> `sudo service maintain-kubeusers restart` on tools-k8s-master-01 [tools]
2019-08-30 §
16:54 <phamhi> restart maintain-kuberusers service in tools-k8s-master-01 [tools]
16:21 <bstorm_> depooling tools-sgewebgrid-lighttpd-0923 to reboot -- high iowait likely from NFS mounts [tools]
2019-08-29 §
22:18 <bd808> Finished building new stretch Docker images for Toolforge Kubernetes use [tools]
22:06 <bd808> Starting process of building new stretch Docker images for Toolforge Kubernetes use [tools]
22:05 <bd808> Jessie Docker image rebuild complete [tools]
21:31 <bd808> Starting process of building new jessie Docker images for Toolforge Kubernetes use [tools]
2019-08-27 §
19:10 <bd808> Restarted maintain-kubeusers after complaint on irc. It was stuck in limbo again [tools]
2019-08-26 §
21:48 <bstorm_> repooled tools-sgewebgrid-generic-0902, tools-sgewebgrid-lighttpd-0902, tools-sgewebgrid-lighttpd-0903 and tools-sgeexec-0905 [tools]
2019-08-18 §
08:11 <arturo> restart maintain-kuberusers service in tools-k8s-master-01 [tools]
2019-08-17 §
10:56 <arturo> force-reboot tools-worker-1006. Is completely stuck [tools]
2019-08-15 §
15:32 <jeh> upgraded jobutils debian package to 1.38 T229551 [tools]
09:22 <arturo> restart maintain-kubeusers service in tools-k8s-master-01 because some tools were missing their namespaces [tools]
2019-08-13 §
22:00 <bstorm_> truncated exim paniclog on tools-sgecron-01 because it was being spammy [tools]
13:41 <jeh> Set icingia downtime for toolschecker labs showmount T229448 [tools]
2019-08-12 §
16:08 <phamhi> updated prometheus-node-exporter from 0.14.0~git20170523-1 to 0.17.0+ds-3 in tools-worker-[1030-1040] nodes (T230147) [tools]
2019-08-08 §
19:25 <jeh> restarting tools-sgewebgrid-lighttpd-0915 T230157 [tools]
2019-08-07 §
19:07 <bd808> Disassociated SUL and Phabricator accounts from user Lophi (T229713) [tools]
2019-08-06 §
16:18 <arturo> add phamhi as user/projectadmin (T228942) and delete hpham [tools]
15:58 <arturo> add hpham as user/projectadmin (T228942) [tools]
13:43 <jeh> disabling puppet on tools-checker-03 while testing nginx timeouts T221301 [tools]
2019-08-05 §
22:49 <bstorm_> launching tools-worker-1040 [tools]
20:36 <andrewbogott> rebooting oom tools-worker-1026 [tools]
16:10 <jeh> `tools-k8s-master-01: systemctl restart maintain-kubeusers` T229846 [tools]
09:39 <arturo> `root@tools-checker-03:~# toolscheckerctl restart` again (T229787) [tools]
09:30 <arturo> `root@tools-checker-03:~# toolscheckerctl restart` (T229787) [tools]
2019-08-02 §
14:00 <andrewbogott_> rebooting tools-worker-1022 as it is unresponsive [tools]
2019-07-31 §
18:07 <bstorm_> drained tools-worker-1015/05/03/17 to rebalance load [tools]
17:41 <bstorm_> drained tools-worker-1025 and 1026 to rebalance load [tools]
17:32 <bstorm_> drained tools-worker-1028 to rebalance load [tools]
17:29 <bstorm_> drained tools-worker-1008 to rebalance load [tools]
17:23 <bstorm_> drained tools-worker-1021 to rebalance load [tools]
17:17 <bstorm_> drained tools-worker-1007 to rebalance load [tools]
17:07 <bstorm_> drained tools-worker-1004 to rebalance load [tools]
16:27 <andrewbogott> moving tools-static-12 to cloudvirt1018 [tools]
15:33 <bstorm_> T228573 spinning up 5 worker nodes for kubernetes cluster (tools-worker-1035-9) [tools]
2019-07-27 §
23:00 <zhuyifei1999_> a past probably related ticket: T194859 [tools]
22:57 <zhuyifei1999_> maintain-kubeusers seems stuck. Traceback: https://phabricator.wikimedia.org/P8812, core dump: /root/core.17898. Restarting [tools]
2019-07-26 §
17:39 <bstorm_> restarted maintain-kubeusers because it was suspiciously tardy and quiet [tools]
17:14 <bstorm_> drained tools-worker-1013.tools.eqiad.wmflabs to rebalance load [tools]
17:09 <bstorm_> draining tools-worker-1020.tools.eqiad.wmflabs to rebalance load [tools]
16:32 <bstorm_> created tools-worker-1034 - T228573 [tools]
15:57 <bstorm_> created tools-worker-1032 and 1033 - T228573 [tools]
15:54 <bstorm_> created tools-worker-1031 - T228573 [tools]
2019-07-25 §
22:01 <bstorm_> T228573 created tools-worker-1030 [tools]
21:22 <jeh> rebooting tools-worker-1016 unresponsive [tools]
2019-07-24 §
10:14 <arturo> reallocating tools-puppetmaster-01 from cloudvirt1027 to cloudvirt1028 (T227539) [tools]
10:12 <arturo> reallocating tools-docker-registry-04 from cloudvirt1027 to cloudvirt1028 (T227539) [tools]