3351-3400 of 3665 results (31ms)
2016-05-19 §
21:07 <bd808> deployed jobutils 1.13 on bastions; now with '-l release=...' validation! [tools]
15:43 <YuviPanda> rebooting all tools worker instances [tools]
13:12 <chasemp> reboot tools-exec-1220 stuck in state of unresponsivenss [tools]
2016-05-13 §
00:40 <YuviPanda> cleared all queues that were in error state [tools]
2016-05-12 §
22:59 <YuviPanda> restart tools-worker-1004 to attempt bringing it back up [tools]
22:59 <YuviPanda> deploy k8s 1.2.4wmf1 on all proxy nodes [tools]
22:58 <YuviPanda> deploy k8s on all worker nodes [tools]
22:46 <YuviPanda> deploy k8s master for 1.2.4wmf1 [tools]
2016-05-10 §
04:25 <bd808> Added role::package::builder to tools-services-01 [tools]
2016-05-09 §
04:33 <YuviPanda> reboot tools-worker-1004, lots of ksoftirqd stuckness despite no actual containers running [tools]
2016-05-08 §
07:06 <YuviPanda> restarted admin tool [tools]
2016-05-05 §
13:11 <godog> cherry-pick https://gerrit.wikimedia.org/r/#/c/280652/ on puppetmaster [tools]
2016-04-28 §
04:15 <YuviPanda> delete half of the trusty webservice jobs [tools]
04:00 <YuviPanda> deleted all precise webservice jobs, waiting for webservicemonitor to bring them back up [tools]
2016-04-24 §
12:22 <YuviPanda> force deleted job 5435259 from pbbot per PeterBowman [tools]
2016-04-11 §
14:20 <andrewbogott> moving tools-bastion-mtemp to labvirt1009 [tools]
2016-04-06 §
15:20 <bd808> Removed local hack for T131906 from tools-puppetmaster-01 [tools]
2016-03-30 §
15:50 <andrewbogott> rebooting tools-proxy-01 in hopes of clearing some bad caches [tools]
2016-03-28 §
20:51 <yuvipanda> lifted RAM quota from 900Gigs to 1TB?! [tools]
20:30 <chasemp> change perm grant files from create-dbusers for chmod 400 chat chattr +i [tools]
2016-03-18 §
15:47 <chasemp> had to kill stalkboten as it was logging constant errors filling logs to the tune of hundreds of gigs [tools]
15:36 <chasemp> cleanup huge log collection for broken bot: /srv/project/tools/project/betacommand-dev/tspywiki/irc/logs# rm -fR SpamBotLog.log\.* [tools]
2016-03-11 §
20:57 <mutante> reverted font changes - puppet runs recovering [tools]
20:37 <mutante> more puppet issues due to font dependencies on trusty, on it [tools]
19:39 <mutante> should a tools-exec server be influenced by font packages on an mw appserver? [tools]
19:39 <mutante> fixed puppet runs on tools-exec (gerrit 276792) [tools]
2016-03-02 §
14:56 <chasemp> qdel 3956069 and 3758653 for abusing auth [tools]
2016-02-28 §
20:08 <bd808> Removed unwanted NFS mounts from tools-elastic-01.tools.eqiad.wmflabs [tools]
2016-02-26 §
19:08 <bd808> Upgraded Elasticsearch on tools-elastic-0[123] to 1.7.5 [tools]
2016-02-24 §
19:46 <chasemp> runonce deployed for https://gerrit.wikimedia.org/r/#/c/272891/ [tools]
2016-02-19 §
15:58 <chasemp> rerollout tools nfs shaping pilot for sanity in anticipation of formalization [tools]
09:21 <_joe_> killed cluebot3 instance on tools-exec-1207, writing 20 M/s to the error log [tools]
00:50 <yuvipanda> failover services to services-02 [tools]
2016-02-18 §
22:57 <valhallasw`cloud> restarted gridengine-master on tools-grid-master, otherwise all webservices will stay down [tools]
20:37 <yuvipanda> failover proxy back to tools-proxy-01 [tools]
19:46 <chasemp> repool labvirt1003 and depool labvirt1004 [tools]
18:19 <chasemp> draining nodes from labvirt1001 [tools]
2016-02-16 §
21:33 <chasemp> reboot of bastion-1002 [tools]
2016-02-12 §
19:56 <chasemp> nfs traffic shaping pilot round 2 [tools]
2016-02-05 §
22:01 <chasemp> throttle some vm nfs write speeds [tools]
2016-02-03 §
03:00 <YuviPanda> upgraded flannel on all hosts running it [tools]
2016-01-29 §
21:25 <YuviPanda> restarted image-resize-calc manually, no service.manifest file [tools]
2016-01-27 §
23:07 <YuviPanda> removed all members of templatetiger, added self instead, removed active shell sessions [tools]
20:24 <chasemp> master stop, truncate accounting log to accounting.01272016, master start [tools]
19:34 <chasemp> master start grid master [tools]
19:23 <chasemp> stopped master [tools]
19:11 <YuviPanda> depooled tools-webgrid-1405 to prep for restart, lots of stuck processes [tools]
18:29 <valhallasw`cloud> job 2551539 is ifttt, which is also running as 2700629. Killing 2551539 . [tools]
18:26 <valhallasw`cloud> messages repeatedly reports "01/27/2016 18:26:17|worker|tools-grid-master|E|execd@tools-webgrid-generic-1405.tools.eqiad.wmflabs reports running job (2551539.1/master) in queue "webgrid-generic@tools-webgrid-generic-1405.tools.eqiad.wmflabs" that was not supposed to be there - killing". SSH'ing there to investigate [tools]
18:24 <valhallasw`cloud> 'sleep' test job also seems to work without issues [tools]