Search

On or before date

go

2015-08-18 §
13:57	<valhallasw`cloud>	same issue seems to happen with the other hosts: tools-exec-1401.tools.eqiad.wmflabs vs tools-exec-1401.eqiad.wmflabs and tools-exec-catscan.tools.eqiad.wmflabs vs tools-exec-catscan.eqiad.wmflabs.	[tools]
13:55	<valhallasw`cloud>	no, wait, that's ''tools-webgrid-lighttpd-1411.eqiad.wmflabs'', not the actual host ''tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs''. We should fix that dns mess as well.	[tools]
13:54	<valhallasw`cloud>	tried to restart gridengine-exec on tools-exec-1401, no effect. tools-webgrid-lighttpd-1411 also just went into 'au' state.	[tools]
13:47	<valhallasw`cloud>	that brought tools-exec-1403, tools-exec-1406 and tools-webgrid-generic-1402 back up, tools-exec-1401 and tools-exec-catscan are still in 'au' state	[tools]
13:46	<valhallasw`cloud>	starting gridengine-exec on hosts with queues in 'au' (=alarm, unknown) state using <code>for i in $(qstat -f -xml | grep "<state>au" -B 6 | grep "<name>" | cut -d'@' -f2 | cut -d. -f1); do echo $i; ssh $i sudo service gridengine-exec start; done</code>	[tools]
08:37	<valhallasw`cloud>	sudo service gridengine-exec start on tools-webgrid-lighttpd-1404.eqiad.wmflabs" tools-webgrid-lighttpd-1406.eqiad.wmflabs" tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs"	[tools]
08:33	<valhallasw`cloud>	tools-webgrid-lighttpd-1403.eqiad.wmflabs, tools-webgrid-lighttpd-1404.eqiad.wmflabs and tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs are all broken (queue dropped because it is temporarily not available)	[tools]
08:30	<valhallasw`cloud>	hostname mismatch: host is called tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs in config, but it was named tools-webgrid-lighttpd-1411.eqiad.wmflabs in the hostgroup config	[tools]
08:21	<valhallasw`cloud>	still sudo qmod -e "*@tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs" -> invalid queue "*@tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs"	[tools]
08:20	<valhallasw`cloud>	sudo qconf -mhgrp "@webgrid", added tools-webgrid-lighttpd-1411.eqiad.wmflabs	[tools]
08:14	<valhallasw`cloud>	and the hostgroup @webgrid doesn't even exist? (╯°□°）╯︵ ┻━┻	[tools]
08:10	<valhallasw`cloud>	/var/lib/gridengine/etc/queues/webgrid-lighttpd does not seem to be the correct configuration as the current config refers to '@webgrid' as host list.	[tools]
08:07	<valhallasw`cloud>	sudo qconf -Ae /var/lib/gridengine/etc/exechosts/tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs -> root@tools-bastion-01.eqiad.wmflabs added "tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs" to exechost list	[tools]
08:06	<valhallasw`cloud>	ok, success. /var/lib/gridengine/etc/exechosts/tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs now exists. Do I still have to add it manually to the grid? I suppose so.	[tools]
08:04	<valhallasw`cloud>	installing packages from /data/project/.system/deb-trusty seems to fail. sudo apt-get update helps.	[tools]
08:00	<valhallasw`cloud>	running puppet agent -tv again	[tools]
07:55	<valhallasw`cloud>	argh. Disabling toollabs::node::web::generic again and enabling toollabs::node::web::lighttpd	[tools]
07:54	<valhallasw`cloud>	various issues such as Error: /Stage[main]/Gridengine::Submit_host/File[/var/lib/gridengine/default/common/accounting]/ensure: change from absent to link failed: Could not set 'link' on ensure: No such file or directory - /var/lib/gridengine/default/common at 17:/etc/puppet/modules/gridengine/manifests/submit_host.pp; probably an ordering issue in	[tools]
07:53	<valhallasw`cloud>	Setting up adminbot (1.7.8) ... chmod: cannot access '/usr/lib/adminbot/README': No such file or directory --- ran sudo touch /usr/lib/adminbot/README	[tools]
07:37	<valhallasw`cloud>	applying role::labs::tools::compute and toollabs::node::web::generic to \tools-webgrid-lighttpd-1411	[tools]
07:31	<valhallasw`cloud>	reading puppet suggests I should qconf -ah /var/lib/gridengine/etc/exechosts/tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs but that file is missing?	[tools]
07:26	<valhallasw`cloud>	andrewbogott built tools-webgrid-lighttpd-1411 yesterday but it's not actually added as exec host. Trying to figure out how to do that...	[tools]
2015-08-17 §
16:17	<andrewbogott>	disable queues for tools-exec-1205 tools-exec-1207 tools-exec-1208 tools-exec-140 tools-exec-1404 tools-exec-1409 tools-exec-1410 tools-exec-catscan tools-web-static-01 tools-webgrid-lighttpd-1201 tools-webgrid-lighttpd-1205 tools-webgrid lighttpd-1206 tools-webgrid-lighttpd-1406 tools-webproxy-02	[tools]
15:33	<andrewbogott>	re-enabling the queue on tools-exec-1211 tools-exec-1212 tools-exec-1215 tools-exec-1403 tools-exec-1406 tools-master tools-shadow tools-webgrid-generic-1402 tools-webgrid-lighttpd-1203 tools-webgrid-lighttpd-1208 tools-webgrid-lighttpd-1403 tools-webgrid-lighttpd-1404 tools-webproxy-01	[tools]
14:50	<andrewbogott>	killing remaining jobs on tools-exec-1211 tools-exec-1212 tools-exec-1215 tools-exec-1403 tools-exec-1406 tools-master tools-shadow tools-webgrid-generic-1402 tools-webgrid-lighttpd-1203 tools-webgrid-lighttpd-1208 tools-webgrid-lighttpd-1403 tools-webgrid-lighttpd-1404 tools-webproxy-01	[tools]
2015-08-15 §
05:14	<andrewbogott>	resumed tools-exec-gift, seems not to have been the culprit	[tools]
05:08	<andrewbogott>	suspending tools-exec-gift, just for a moment...	[tools]
2015-08-14 §
17:21	<andrewbogott>	disabling grid jobqueue for tools-exec-1211 tools-exec-1212 tools-exec-1215 tools-exec-1403 tools-exec-1406 tools-master tools-shadow tools-webgrid-generic-1402 tools-webgrid-lighttpd-1203 tools-webgrid-lighttpd-1208 tools-webgrid-lighttpd-1403 tools-webgrid-lighttpd-1404 tools-webproxy-01 in anticipation of monday reboot of labvirt1004	[tools]
15:20	<andrewbogott>	Adding back to the grid engine queue: tools-exec-1216 tools-exec-1219 tools-exec-1407 tools-mail tools-services-02 tools-webgrid-generic-1401 tools-webgrid-lighttpd-1202 tools-webgrid-lighttpd-1207 tools-webgrid-lighttpd-1210 tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1407	[tools]
14:43	<andrewbogott>	killing remaining jobs on tools-exec-1216 tools-exec-1219 tools-exec-1407 tools-mail tools-services-02 tools-webgrid-generic-1401 tools-webgrid-lighttpd-1202 tools-webgrid-lighttpd-1207 tools-webgrid-lighttpd-1210 tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1407	[tools]
2015-08-13 §
18:51	<valhallasw`cloud>	which was resolved by scfc earlier	[tools]
18:50	<valhallasw`cloud>	tools-exec-1201/Puppet staleness was critical due to an agent lock (Ignoring stale puppet agent lock for pid <br> Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists))	[tools]
16:44	<andrewbogott>	disabling job queue for tools-exec-1216 tools-exec-1219 tools-exec-1407 tools-mail tools-services-02 tools-webgrid-generic-1401 tools-webgrid-lighttpd-1202 tools-webgrid-lighttpd-1207 tools-webgrid-lighttpd-1210 tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1407	[tools]
14:48	<andrewbogott>	and tools-webgrid-lighttpd-1408	[tools]
14:48	<andrewbogott>	rescheduling (and in some cases killing) jobs on tools-exec-1203 tools-exec-1210 tools-exec-1214 tools-exec-1402 tools-exec-1405 tools-exec-gift tools-services-01 tools-web-static-02 tools-webgrid-generic-1403 tools-webgrid-lighttpd-1204 tools-webgrid-lighttpd-1209 tools-webgrid-lighttpd-1401 tools-webgrid-lighttpd-1405	[tools]
2015-08-12 §
16:05	<andrewbogott>	depooling tools-exec-1203 tools-exec-1210 tools-exec-1214 tools-exec-1402 tools-exec-1405 tools-exec-gift tools-services-01 tools-web-static-02 tools-webgrid-generic-1403 tools-webgrid-lighttpd-1204 tools-webgrid-lighttpd-1209 tools-webgrid-lighttpd-1401 tools-webgrid-lighttpd-1405 tools-webgrid-lighttpd-1408	[tools]
14:41	<andrewbogott>	forcing reschedule of jobs on tools-exec-1201 tools-exec-1202 tools-exec-1204 tools-exec-1206 tools-exec-1209 tools-exec-1213 tools-exec-1217 tools-exec-1218 tools-exec-1408 tools-webgrid-generic-1404 tools-webgrid-lighttpd-1409 tools-webgrid-lighttpd-1410	[tools]
2015-08-11 §
18:17	<andrewbogott>	depooling tools-exec-1201 tools-exec-1202 tools-exec-1204 tools-exec-1206 tools-exec-1209 tools-exec-1213 tools-exec-1217 tools-exec-1218 tools-exec-1408 tools-webgrid-generic-1404 tools-webgrid-lighttpd-1409 tools-webgrid-lighttpd-1410 in anticipation of labvirt1001 reboot tomorrow	[tools]
2015-08-03 §
19:13	<andrewbogott>	deleted tools-static-01	[tools]
2015-08-01 §
18:09	<andrewbogott>	depooling/rebooting tools-webgrid-lighttpd-1407 because it’s unable to fork	[tools]
2015-07-30 §
15:00	<andrewbogott>	rebooting tools-bastion-01 aka tools-login	[tools]
2015-07-29 §
23:43	<andrewbogott>	draining, rebooting tools-webgrid-lighttpd-1408	[tools]
20:11	<andrewbogott>	rebooting tools-webgrid-lighttpd-1404	[tools]
2015-07-28 §
17:49	<valhallasw`cloud>	Jobs were drained at 19:43, but this did not decreade he rate, which is still at ~50k/minute. Now running "sysctl -w sunrpc.nfs_debug=1023 && sleep 2 && sysctl -w sunrpc.nfs_debug=0" which hopefully doesn't kill the server	[tools]
17:43	<valhallasw`cloud>	rescheduled all webservice jobs on tools-webgrid-lighttpd-1401.eqiad.wmflabs, server is now empty	[tools]
17:16	<valhallasw`cloud>	disabled queue "webgrid-lighttpd@tools-webgrid-lighttpd-1401.eqiad.wmflabs"	[tools]
02:06	<YuviPanda>	removed pacct files from tools-bastion-01	[tools]
2015-07-27 §
21:27	<valhallasw`cloud>	turned off process accounting on tools-login while we try to find the root cause of [[phab:T107052]]: <pre>accton off</pre>	[tools]