151-200 of 3081 results (12ms)
2021-03-18 §
03:37 <bstorm> deleted a massive number of stuck jobs that misfired from the cron server [tools]
03:35 <bstorm> rebooting tools-sgecron-01 to try to clear up the ldap-related errors coming out of it [tools]
01:46 <bstorm> killed the toolschecker cron job, which had an LDAP error, and ran it again by hand [tools]
2021-03-17 §
20:57 <bstorm> deployed changes to rbac for kubernetes to add kubectl top access for tools [tools]
20:26 <andrewbogott> moving tools-elastic-3 to cloudvirt1034; two elastic nodes shouldn't be on the same hv [tools]
2021-03-16 §
16:31 <arturo> installing jobutils and misctools 1.41 [tools]
15:55 <bstorm> deleted a bunch of messed up grid jobs (9989481,8813,81682,86317,122602,122623,583621,606945,606999) [tools]
12:32 <arturo> add packages jobutils / misctools v1.41 to {stretch,buster}-tools aptly repository in tools-sge-services-03 [tools]
2021-03-12 §
23:13 <bstorm> cleared error state for all grid queues [tools]
2021-03-11 §
17:40 <bstorm> deployed metrics-server:0.4.1 to kubernetes [tools]
16:21 <bstorm> add jobutils 1.40 and misctools 1.40 to stretch-tools [tools]
13:11 <arturo> add misctools 1.37 to buster-tools|toolsbeta aptly repo for T275865 [tools]
13:10 <arturo> add jobutils 1.40 to buster-tools aptly repo for T275865 [tools]
2021-03-10 §
10:56 <arturo> briefly stopped VM tools-k8s-etcd-7 to disable VMX cpu flag [tools]
2021-03-09 §
13:31 <arturo> hard-reboot tools-docker-registry-04 because issues related to T276922 [tools]
12:34 <arturo> briefly rebooting VM tools-docker-registry-04, we need to reboot the hypervisor cloudvirt1038 and failed to migrate away [tools]
2021-03-05 §
12:30 <arturo> started tools-redis-1004 again [tools]
12:22 <arturo> stop tools-redis-1004 to ease draining of cloudvirt1035 [tools]
2021-03-04 §
11:25 <arturo> rebooted tools-sgewebgrid-generic-0901, repool it again [tools]
09:57 <arturo> depool tools-sgewebgrid-generic-0901 to reboot VM. It was stuck in MIGRATING state when draining cloudvirt1022 [tools]
2021-03-03 §
15:17 <arturo> shutting down tools-sgebastion-07 in an attempt to fix nova state and finish hypervisor migration [tools]
15:11 <arturo> tools-sgebastion-07 triggered a neutron exception (unauthorized) while being live-migrated from cloudvirt1021 to 1029. Resetting nova state with `nova reset-state bd685d48-1011-404e-a755-372f6022f345 --active` and try again [tools]
14:48 <arturo> killed pywikibot instance running in tools-sgebastion-07 by user msyn [tools]
2021-03-02 §
15:23 <bstorm> depooling tools-sgewebgrid-lighttpd-0914.tools.eqiad.wmflabs for reboot. It isn't communicating right [tools]
15:22 <bstorm> cleared queue error states...will need to keep a better eye on what's causing those [tools]
2021-02-27 §
02:23 <bstorm> deployed typo fix to maintain-kubeusers in an innocent effort to make the weekend better T275910 [tools]
02:00 <bstorm> running a script to repair the dumps mount in all podpresets T275371 [tools]
2021-02-26 §
22:04 <bstorm> cleaned up grid jobs 1230666,1908277,1908299,2441500,2441513 [tools]
21:27 <bstorm> hard rebooting tools-sgeexec-0947 [tools]
21:21 <bstorm> hard rebooting tools-sgeexec-0952.tools.eqiad.wmflabs [tools]
20:01 <bd808> Deleted csr in strange state for tool-ores-inspect [tools]
2021-02-24 §
18:30 <bd808> `sudo wmcs-openstack role remove --user zfilipin --project tools user` T267313 [tools]
01:04 <bstorm> hard rebooting tools-k8s-worker-76 because it's in a sorry state [tools]
2021-02-23 §
23:11 <bstorm> draining a bunch of k8s workers to clean up after dumps changes T272397 [tools]
23:06 <bstorm> draining tools-k8s-worker-55 to clean up after dumps changes T272397 [tools]
2021-02-22 §
20:40 <bstorm> repooled tools-sgeexec-0918.tools.eqiad.wmflabs [tools]
19:09 <bstorm> hard rebooted tools-sgeexec-0918 from openstack T275411 [tools]
19:07 <bstorm> shutting down tools-sgeexec-0918 with the VM's command line (not libvirt directly yet) T275411 [tools]
19:05 <bstorm> shutting down tools-sgeexec-0918 (with openstack to see what happens) T275411 [tools]
19:03 <bstorm> depooled tools-sgeexec-0918 T275411 [tools]
18:56 <bstorm> deleted job 1962508 from the grid to clear it up T275301 [tools]
16:58 <bstorm> cleared error state on several grid queues [tools]
2021-02-19 §
12:31 <arturo> deploying new version of toolforge ingress admission controller [tools]
2021-02-17 §
21:26 <bstorm> deleted tools-puppetdb-01 since it is unused at this time (and undersized anyway) [tools]
2021-02-04 §
16:27 <bstorm> rebooting tools-package-builder-02 [tools]
2021-01-26 §
16:27 <bd808> Hard reboot of tools-sgeexec-0906 via Horizon for T272978 [tools]
2021-01-22 §
09:59 <dcaro> added the record redis.svc.tools.eqiad1.wikimedia.cloud pointing to tools-redis1003 (T272679) [tools]
2021-01-21 §
23:58 <bstorm> deployed new maintain-kubeusers to tools T271847 [tools]
2021-01-19 §
22:57 <bstorm> truncated 75GB error log /data/project/robokobot/virgule.err T272247 [tools]
22:48 <bstorm> truncated 100GB error log /data/project/magnus-toolserver/error.log T272247 [tools]