251-300 of 943 results (15ms)
2019-10-23 §
20:00 <phamhi> Rebuilding all jessie and stretch docker images to pick up toollabs-webservice 0.47 (T233347) [tools]
12:09 <phamhi> Deployed toollabs-webservice 0.47 to buster-tools and stretch-tools (T233347) [tools]
09:13 <arturo> 9 tools-sgeexec nodes and 6 other related VMs are down because hypervisor is rebooting [tools]
2019-10-21 §
17:32 <phamhi> Rebuilding all jessie and stretch docker images to pick up toollabs-webservice 0.46 [tools]
2019-10-18 §
22:15 <bd808> Rescheduled continuous jobs away from tools-sgeexec-0904 because of high system load [tools]
22:09 <bd808> Cleared error state of webgrid-generic@tools-sgewebgrid-generic-0901, webgrid-lighttpd@tools-sgewebgrid-lighttpd-09{12,15,19,20,26} [tools]
21:29 <bd808> Rescheduled all grid engine webservice jobs (T217815) [tools]
2019-10-16 §
16:21 <phamhi> Deployed toollabs-webservice 0.46 to buster-tools and stretch-tools (T218461) [tools]
09:17 <arturo> due to the reboot of cloudvirt1029, several sgeexec nodes (8) are offline, also sgewebgrid-lighttpd (8) and tools-worker (3) and the main toolforge proxy (tools-proxy-03) [tools]
2019-10-11 §
11:55 <arturo> create tools-test-proxy-01 VM for testing T235059 and a puppet prefix for it [tools]
10:53 <arturo> added kubernetes-node_1.4.6-7_amd64.deb to buster-tools and buster-toolsbeta (aptly) for T235059 [tools]
10:51 <arturo> added docker-engine_1.12.6-0~debian-jessie_amd64.deb to buster-tools and buster-toolsbeta (aptly) for T235059 [tools]
10:46 <arturo> added logster_0.0.10-2~jessie1_all.deb to buster-tools and buster-toolsbeta (aptly) for T235059 [tools]
2019-10-10 §
02:33 <bd808> Rebooting tools-sgewebgrid-lighttpd-0903. Instance hung. [tools]
2019-10-09 §
14:46 <phamhi> drained and cordoned tools-worker-1029 after status reset on reboot [tools]
2019-10-08 §
19:34 <bstorm_> drained tools-worker-1009 and then 1014 for rebalancing [tools]
19:27 <bstorm_> drained tools-worker-1005 for rebalancing (and put these back in service as I went) [tools]
19:24 <bstorm_> drained tools-worker-1003 and 1009 for rebalancing [tools]
2019-10-07 §
20:17 <bd808> Dropped backlog of messages for delivery to tools.usrd-tools [tools]
20:16 <bd808> Dropped backlog of messages for delivery to tools.mix-n-match [tools]
20:13 <bd808> Dropped backlog of frozen messages for delivery (240 dropped) [tools]
19:00 <Krenair> reboot tools-static-12 tools-docker-registry-04 and tools-clushmaster-02 due to NFS stale issue [tools]
17:35 <phamhi> drained and uncordoned tools-worker-100[1-5] [tools]
17:01 <bstorm_> rebooting tools-sgegrid-master and tools-sgegrid-shadow 😭 [tools]
16:49 <bstorm_> rebooting tools-sgeexec-0901 and tools-sgeexec-0909/10/11 [tools]
16:46 <bd808> `sudo shutdown -r now` for tools-sgebastion-08 [tools]
16:39 <bd808> `sudo service nslcd restart` on tools-sgebastion-08 [tools]
2019-10-04 §
21:43 <bd808> `sudo exec-manage repool tools-sgeexec-0923.tools.eqiad.wmflabs` [tools]
21:26 <bd808> Rebooting tools-sgeexec-0923 after lots of messing about with a broken update-initramfs build [tools]
20:35 <bd808> Manually running `/usr/bin/python3 /usr/bin/unattended-upgrade` on tools-sgeexec-0923 [tools]
20:33 <bd808> Killed 2 /usr/bin/unattended-upgrade procs on tools-sgeexec-0923 that seemed stuck [tools]
2019-09-27 §
16:59 <bd808> Set "profile::rsyslog::kafka_shipper::kafka_brokers: []" in tools-elastic prefix puppet [tools]
00:40 <bstorm_> depooled and rebooted tools-sgewebgrid-lighttpd-0927 [tools]
2019-09-23 §
16:58 <bstorm_> deployed tools-manifest 0.20 and restarted webservicemonitor [tools]
06:01 <bd808> Restarted maintain-dbusers process on labstore1004. (T233530) [tools]
2019-09-09 §
22:44 <bstorm_> uncordoned tools-worker-1030 and tools-worker-1038 [tools]
2019-09-06 §
15:11 <bd808> `sudo kill -9 10635` on tools-k8s-master-01 (T194859) [tools]
2019-09-05 §
21:02 <bd808> Enabled Puppet on tools-docker-registry-03 and forced puppet run (T232135) [tools]
18:13 <bd808> Disabled Puppet on tools-docker-registry-03 to investigate docker-registry issue (no phab task yet) [tools]
2019-08-29 §
22:18 <bd808> Finished building new stretch Docker images for Toolforge Kubernetes use [tools]
22:06 <bd808> Starting process of building new stretch Docker images for Toolforge Kubernetes use [tools]
22:05 <bd808> Jessie Docker image rebuild complete [tools]
21:31 <bd808> Starting process of building new jessie Docker images for Toolforge Kubernetes use [tools]
2019-08-27 §
19:10 <bd808> Restarted maintain-kubeusers after complaint on irc. It was stuck in limbo again [tools]
2019-08-26 §
21:48 <bstorm_> repooled tools-sgewebgrid-generic-0902, tools-sgewebgrid-lighttpd-0902, tools-sgewebgrid-lighttpd-0903 and tools-sgeexec-0905 [tools]
2019-08-07 §
19:07 <bd808> Disassociated SUL and Phabricator accounts from user Lophi (T229713) [tools]
2019-08-06 §
16:18 <arturo> add phamhi as user/projectadmin (T228942) and delete hpham [tools]
2019-07-31 §
17:41 <bstorm_> drained tools-worker-1025 and 1026 to rebalance load [tools]
2019-07-26 §
17:39 <bstorm_> restarted maintain-kubeusers because it was suspiciously tardy and quiet [tools]
15:57 <bstorm_> created tools-worker-1032 and 1033 - T228573 [tools]