2951-2974 of 2974 results (17ms)
2015-08-17 §
14:50 <andrewbogott> killing remaining jobs on tools-exec-1211 tools-exec-1212 tools-exec-1215 tools-exec-1403 tools-exec-1406 tools-master tools-shadow tools-webgrid-generic-1402 tools-webgrid-lighttpd-1203 tools-webgrid-lighttpd-1208 tools-webgrid-lighttpd-1403 tools-webgrid-lighttpd-1404 tools-webproxy-01 [tools]
2015-08-15 §
05:14 <andrewbogott> resumed tools-exec-gift, seems not to have been the culprit [tools]
05:08 <andrewbogott> suspending tools-exec-gift, just for a moment... [tools]
2015-08-14 §
17:21 <andrewbogott> disabling grid jobqueue for tools-exec-1211 tools-exec-1212 tools-exec-1215 tools-exec-1403 tools-exec-1406 tools-master tools-shadow tools-webgrid-generic-1402 tools-webgrid-lighttpd-1203 tools-webgrid-lighttpd-1208 tools-webgrid-lighttpd-1403 tools-webgrid-lighttpd-1404 tools-webproxy-01 in anticipation of monday reboot of labvirt1004 [tools]
15:20 <andrewbogott> Adding back to the grid engine queue: tools-exec-1216 tools-exec-1219 tools-exec-1407 tools-mail tools-services-02 tools-webgrid-generic-1401 tools-webgrid-lighttpd-1202 tools-webgrid-lighttpd-1207 tools-webgrid-lighttpd-1210 tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1407 [tools]
14:43 <andrewbogott> killing remaining jobs on tools-exec-1216 tools-exec-1219 tools-exec-1407 tools-mail tools-services-02 tools-webgrid-generic-1401 tools-webgrid-lighttpd-1202 tools-webgrid-lighttpd-1207 tools-webgrid-lighttpd-1210 tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1407 [tools]
2015-08-13 §
18:51 <valhallasw`cloud> which was resolved by scfc earlier [tools]
18:50 <valhallasw`cloud> tools-exec-1201/Puppet staleness was critical due to an agent lock (Ignoring stale puppet agent lock for pid <br> Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists)) [tools]
16:44 <andrewbogott> disabling job queue for tools-exec-1216 tools-exec-1219 tools-exec-1407 tools-mail tools-services-02 tools-webgrid-generic-1401 tools-webgrid-lighttpd-1202 tools-webgrid-lighttpd-1207 tools-webgrid-lighttpd-1210 tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1407 [tools]
14:48 <andrewbogott> and tools-webgrid-lighttpd-1408 [tools]
14:48 <andrewbogott> rescheduling (and in some cases killing) jobs on tools-exec-1203 tools-exec-1210 tools-exec-1214 tools-exec-1402 tools-exec-1405 tools-exec-gift tools-services-01 tools-web-static-02 tools-webgrid-generic-1403 tools-webgrid-lighttpd-1204 tools-webgrid-lighttpd-1209 tools-webgrid-lighttpd-1401 tools-webgrid-lighttpd-1405 [tools]
2015-08-12 §
16:05 <andrewbogott> depooling tools-exec-1203 tools-exec-1210 tools-exec-1214 tools-exec-1402 tools-exec-1405 tools-exec-gift tools-services-01 tools-web-static-02 tools-webgrid-generic-1403 tools-webgrid-lighttpd-1204 tools-webgrid-lighttpd-1209 tools-webgrid-lighttpd-1401 tools-webgrid-lighttpd-1405 tools-webgrid-lighttpd-1408 [tools]
14:41 <andrewbogott> forcing reschedule of jobs on tools-exec-1201 tools-exec-1202 tools-exec-1204 tools-exec-1206 tools-exec-1209 tools-exec-1213 tools-exec-1217 tools-exec-1218 tools-exec-1408 tools-webgrid-generic-1404 tools-webgrid-lighttpd-1409 tools-webgrid-lighttpd-1410 [tools]
2015-08-11 §
18:17 <andrewbogott> depooling tools-exec-1201 tools-exec-1202 tools-exec-1204 tools-exec-1206 tools-exec-1209 tools-exec-1213 tools-exec-1217 tools-exec-1218 tools-exec-1408 tools-webgrid-generic-1404 tools-webgrid-lighttpd-1409 tools-webgrid-lighttpd-1410 in anticipation of labvirt1001 reboot tomorrow [tools]
2015-08-03 §
19:13 <andrewbogott> deleted tools-static-01 [tools]
2015-08-01 §
18:09 <andrewbogott> depooling/rebooting tools-webgrid-lighttpd-1407 because it’s unable to fork [tools]
2015-07-30 §
15:00 <andrewbogott> rebooting tools-bastion-01 aka tools-login [tools]
2015-07-29 §
23:43 <andrewbogott> draining, rebooting tools-webgrid-lighttpd-1408 [tools]
20:11 <andrewbogott> rebooting tools-webgrid-lighttpd-1404 [tools]
2015-07-28 §
17:49 <valhallasw`cloud> Jobs were drained at 19:43, but this did not decreade he rate, which is still at ~50k/minute. Now running "sysctl -w sunrpc.nfs_debug=1023 && sleep 2 && sysctl -w sunrpc.nfs_debug=0" which hopefully doesn't kill the server [tools]
17:43 <valhallasw`cloud> rescheduled all webservice jobs on tools-webgrid-lighttpd-1401.eqiad.wmflabs, server is now empty [tools]
17:16 <valhallasw`cloud> disabled queue "webgrid-lighttpd@tools-webgrid-lighttpd-1401.eqiad.wmflabs" [tools]
02:06 <YuviPanda> removed pacct files from tools-bastion-01 [tools]
2015-07-27 §
21:27 <valhallasw`cloud> turned off process accounting on tools-login while we try to find the root cause of [[phab:T107052]]: <pre>accton off</pre> [tools]