901-943 of 943 results (7ms)
2016-01-08 §
19:46 <chasemp> couldn't get into tools-mail-01 at all and it seemed borked so I rebooted [tools]
2015-12-30 §
02:39 <YuviPanda> remove lbenedix and ebekebe from tools.hcclab [tools]
00:40 <YuviPanda> copied and cleaned out spooldb [tools]
2015-12-28 §
22:34 <chasemp> attempt to unmount nfs volumes on tools-redis-01 to debug but it hands (I am on console and see root at console hang on login) [tools]
22:31 <YuviPanda> disable NFS on tools-redis-1001 and 1002 [tools]
21:32 <YuviPanda> disable puppet on tools-redis-01 and -02 [tools]
2015-12-22 §
18:12 <YuviPanda> upgraded kernel and rebooted tools-proxy-01 [tools]
2015-12-04 §
04:09 <bd808> Cherry-picked https://gerrit.wikimedia.org/r/#/c/256618 to tools-puppetmaster-01 [tools]
2015-12-01 §
21:07 <YuviPanda> added bd808 as admin [tools]
2015-11-20 §
22:01 <Coren> tools-webgrid-lighttpd-1412 tools-webgrid-lighttpd-1413 tools-webgrid-lighttpd-1414 tools-webgrid-lighttpd-1415 done and back in rotation. [tools]
21:45 <Coren> tools-webgrid-lighttpd-1411 tools-webgrid-lighttpd-1211 done and back in rotation. [tools]
21:30 <Coren> tools-webgrid-lighttpd-1410 tools-webgrid-lighttpd-1210 done and back in rotation. [tools]
21:25 <Coren> tools-webgrid-lighttpd-1409 tools-webgrid-lighttpd-1209 done and back in rotation. [tools]
21:13 <Coren> tools-webgrid-lighttpd-1408 tools-webgrid-lighttpd-1208 done and back in rotation. [tools]
20:58 <Coren> tools-webgrid-lighttpd-1407 tools-webgrid-lighttpd-1207 done and back in rotation. [tools]
20:53 <Coren> tools-webgrid-lighttpd-1406 tools-webgrid-lighttpd-1206 done and back in rotation. [tools]
20:41 <Coren> tools-webgrid-lighttpd-1405 tools-webgrid-lighttpd-1205 tools-webgrid-generic-1405 done and back in rotation. [tools]
20:28 <Coren> tools-webgrid-lighttpd-1404 tools-webgrid-lighttpd-1204 tools-webgrid-generic-1404 done and back in rotation. [tools]
19:49 <Coren> done, and putting back in rotation: tools-webgrid-lighttpd-1403 tools-webgrid-lighttpd-1203 tools-webgrid-generic-1403 [tools]
19:15 <Coren> done, and putting back in rotation: tools-webgrid-lighttpd-1402 tools-webgrid-lighttpd-1202 tools-webgrid-generic-1402 [tools]
18:10 <Coren> disabling puppet on the grid nodes listed at https://phabricator.wikimedia.org/P2337 so that the /tmp change in https://gerrit.wikimedia.org/r/#/c/252506/ do not apply early and break services [tools]
2015-11-02 §
22:10 <YuviPanda> created tools-webgrid-lighttpd-1414 and 1415 [tools]
22:04 <YuviPanda> created tools-webgrid-lighttpd-1412 and 1413 [tools]
19:53 <YuviPanda> drained continuous jobs and disabled queues on tools-exec-1203 and tools-exec-1402 [tools]
2015-09-30 §
07:12 <yuvipanda> deleted tools-webproxy-01 and -02, running on proxy-01 and -02 now [tools]
2015-09-23 §
18:22 <valhallasw`cloud> experimenting with https://github.com/jordansissel/fpm on tools-packages, and manually installing packages for that. Noting them here. [tools]
2015-09-07 §
17:57 <YuviPanda> created tools-k8s-master-01 with jessie, will be etcd and kubernetes master [tools]
2015-09-03 §
07:09 <valhallasw`cloud> and just re-running puppet solves the issue. Sigh. [tools]
2015-08-31 §
20:29 <valhallasw`cloud> |sort is not so spread out in terms of affected hosts because a lot of jobs were started on lighttpd-1409 and -1410 around the same time. [tools]
2015-08-30 §
13:23 <valhallasw`cloud> killed wikibugs-backup and grrrit-wm on tools-webproxy-01 [tools]
2015-08-25 §
10:04 <YuviPanda> apply exec node puppet roles to tools-exec-1220 and -1221 [tools]
09:59 <YuviPanda> created tools-exec-1220 and -1221 [tools]
2015-08-24 §
16:37 <valhallasw`cloud> more processes were started, so added a talk page message on [[User:Coet]] (who was starting the processes according to /var/log/auth.log) and using 'write coet' on tools-bastion-01 [tools]
2015-08-20 §
18:42 <valhallasw`cloud> tools-web-static-01 has the same issue, but with different commit ids (because different hostname). No local changes on static-01. The initial merge commit on -01 is 57994c, merging 1e392ab and fc918b8; on -02 it's 511617f, merging a90818c and fc918b8. [tools]
18:39 <valhallasw`cloud> cdnjs on tools-web-static-02 can't pull because it has a dirty working tree, and there's a bunch of weird merge commits. Old commit is c4abeabd3acf614285a40e36538f50655e53b47d, the dirty working tree is changes from http to https in various files [tools]
2015-08-18 §
13:57 <valhallasw`cloud> same issue seems to happen with the other hosts: tools-exec-1401.tools.eqiad.wmflabs vs tools-exec-1401.eqiad.wmflabs and tools-exec-catscan.tools.eqiad.wmflabs vs tools-exec-catscan.eqiad.wmflabs. [tools]
13:47 <valhallasw`cloud> that brought tools-exec-1403, tools-exec-1406 and tools-webgrid-generic-1402 back up, tools-exec-1401 and tools-exec-catscan are still in 'au' state [tools]
08:33 <valhallasw`cloud> tools-webgrid-lighttpd-1403.eqiad.wmflabs, tools-webgrid-lighttpd-1404.eqiad.wmflabs and tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs are all broken (queue dropped because it is temporarily not available) [tools]
08:14 <valhallasw`cloud> and the hostgroup @webgrid doesn't even exist? (╯°□°）╯︵ ┻━┻ [tools]
07:55 <valhallasw`cloud> argh. Disabling toollabs::node::web::generic again and enabling toollabs::node::web::lighttpd [tools]
07:37 <valhallasw`cloud> applying role::labs::tools::compute and toollabs::node::web::generic to \tools-webgrid-lighttpd-1411 [tools]
2015-08-13 §
14:48 <andrewbogott> and tools-webgrid-lighttpd-1408 [tools]
14:48 <andrewbogott> rescheduling (and in some cases killing) jobs on tools-exec-1203 tools-exec-1210 tools-exec-1214 tools-exec-1402 tools-exec-1405 tools-exec-gift tools-services-01 tools-web-static-02 tools-webgrid-generic-1403 tools-webgrid-lighttpd-1204 tools-webgrid-lighttpd-1209 tools-webgrid-lighttpd-1401 tools-webgrid-lighttpd-1405 [tools]