551-600 of 943 results (16ms)
2018-11-16 §
13:36 <gtirloni> rebooted tools-static-12 and tools-static-13 after package upgrades [tools]
2018-11-13 §
17:40 <arturo> remove misctools 1.31 and jobutils 1.30 from the stretch-tools repo (T207970) [tools]
13:22 <arturo> T207970 misctools and jobutils v1.32 are now in both `stretch-tools` and `stretch-toolsbeta` repos in tools-services-01 [tools]
2018-11-08 §
17:58 <arturo> installing jobutils and misctools v1.32 (T207970) [tools]
2018-10-31 §
18:02 <gtirloni> truncated big .err and error.log files [tools]
2018-10-29 §
17:00 <bd808> Ran grid engine orphan process kill script from T153281 [tools]
2018-10-26 §
10:34 <arturo> T207970 added misctools 1.31 and jobutils 1.30 to stretch-tools aptly repo [tools]
10:32 <arturo> T209970 added misctools 1.31 and jobutils 1.30 to stretch-tools aptly repo [tools]
2018-10-19 §
00:29 <andrewbogott> migrating tools-exec-1411 and tools-exec-1410 off of cloudvirt1017 [tools]
2018-10-18 §
19:57 <andrewbogott> moving tools-webgrid-lighttpd-1419, tools-webgrid-lighttpd-1420 and tools-webgrid-lighttpd-1421 to labvirt1009, 1010 and 1011 as part of (gradually) draining labvirt1017 [tools]
2018-10-16 §
15:13 <bd808> (repost for gtirloni) T186571 removed legofan4000 user from project-tools group (leftover from T165624 legofan4000->macfan4000 rename) [tools]
2018-09-08 §
10:35 <gtirloni> restarted cron and truncated /var/log/exim4/paniclog (T196137) [tools]
2018-08-27 §
23:39 <bd808> `# exec-manage repool tools-webgrid-generic-1402.eqiad.wmflabs` T202932 [tools]
23:28 <bd808> Restarted down instance tools-webgrid-generic-1402 & ran apt-upgrade [tools]
2018-08-19 §
09:12 <legoktm> rebuilding python/base k8s images for https://gerrit.wikimedia.org/r/453665 (T202218) [tools]
2018-08-14 §
21:02 <legoktm> rebuilt php7.2 docker images for https://gerrit.wikimedia.org/r/452755 [tools]
01:08 <legoktm> switched tools.coverme and tools.wikiinfo to use PHP 7.2 [tools]
2018-08-13 §
23:31 <legoktm> rebuilding docker images for webservice upgrade [tools]
2018-08-08 §
10:00 <zhuyifei1999_> building & publishing toollabs-webservice 0.40 deb, and all Docker images T156626 T148872 T158244 [tools]
2018-07-30 §
20:33 <bd808> Started rebuilding all Kubernetes Docker images to pick up latest apt updates [tools]
2018-06-30 §
16:40 <zhuyifei1999_> because tools-paws-master-01 was having ~1000 loadavg due to NFS having issues and processes stuck in D state [tools]
2018-06-29 §
17:41 <bd808> Rescheduling continuous jobs away from tools-exec-1408 where load is high [tools]
17:11 <bd808> Rescheduled jobs away from toole-exec-1404 where linkwatcher is currently stealing most of the CPU (T123121) [tools]
16:46 <bd808> Killed orphan tool owned processes running on the job grid. Mostly jembot and wsexport php-cgi processes stuck in deadlock following an OOM. T182070 [tools]
2018-06-28 §
16:40 <andrewbogott> rebooting tools-worker-1012 and tools-worker-1015 to get their nfs mounts unstuck [tools]
2018-06-20 §
15:09 <bd808> Killed orphan processes on webgrid nodes (T182070); most owned by jembot and croptool [tools]
2018-06-08 §
07:46 <arturo> T196137 more rootspam today, restarting again `prometheus-node-exporter` and force rotating exim4 paniclog in 12 nodes [tools]
2018-06-06 §
22:00 <bd808> Scripting a restart of webservice for tools that are still in CrashLoopBackOff state after 2nd attempt (T196589) [tools]
21:10 <bd808> Scripting a restart of webservice for 59 tools that are still in CrashLoopBackOff state after last attempt (P7220) [tools]
20:25 <bd808> Scripting a restart of webservice for 175 tools that are in CrashLoopBackOff state (P7220) [tools]
2018-06-05 §
18:02 <bd808> Forced puppet run on tools-bastion-03 to re-enable logins by dubenben (T196486) [tools]
17:39 <arturo> T196137 clush: delete `prometheus` user and re-create it locally. Then, chown prometheus dirs [tools]
17:38 <bd808> Added grid engine quota to limit user debenben to 2 concurrent jobs (T196486) [tools]
2018-06-03 §
10:19 <zhuyifei1999_> Grid is full. qdel'ed all jobs belonging to tools.dibot except lighttpd, and tools.mbh that has a job name starting 'comm_delin', 'delfilexcl' T195834 [tools]
2018-05-28 §
12:09 <arturo> T194665 adding mono packages to apt.wikimedia.org for jessie-wikimedia and stretch-wikimedia [tools]
2018-05-18 §
16:36 <bd808> Restarted bigbrother on tools-services-02 [tools]
2018-05-07 §
21:02 <zhuyifei1999_> re-building all docker images T190893 [tools]
20:48 <zhuyifei1999_> building, signing, and publishing toollabs-webservice 0.39 T190893 [tools]
2018-04-22 §
13:07 <bd808> Kill orphan php-cgi processes across the job grid via clush -w @exec -w @webgrid -b 'ps axwo user:20,ppid,pid,cmd | grep -E " 1 " | grep php-cgi | xargs sudo kill -9'` [tools]
2018-03-29 §
19:56 <chicocvenancio> several interactive jobs running in bastion-03. I am writing to connected users and will kill the jobs once done [tools]
2018-03-26 §
21:34 <bd808> clush -w @exec -w @webgrid -b 'sudo find /tmp -type f -atime +1 -delete' [tools]
2018-03-23 §
23:26 <bd808> clush -w @exec -w @webgrid -b 'sudo find /tmp -type f -atime +1 -delete' [tools]
19:43 <bd808> tools-proxy-* Forced puppet run to apply https://gerrit.wikimedia.org/r/#/c/421472/ [tools]
2018-03-22 §
22:04 <bd808> Forced puppet run on tools-proxy-02 for T130748 [tools]
21:52 <bd808> Forced puppet run on tools-proxy-01 for T130748 [tools]
21:48 <bd808> Disabled puppet on tools-proxy-* for https://gerrit.wikimedia.org/r/#/c/420619/ rollout [tools]
03:50 <bd808> clush -w @exec -w @webgrid -b 'sudo find /tmp -type f -atime +1 -delete' [tools]
2018-03-21 §
17:50 <bd808> Cleaned up stale /project/.system/bigbrother.scoreboard.* files from labstore1004 [tools]
01:09 <bd808> Deleting /tmp files owned by tools.wsexport with -mtime +2 across grid (T190185) [tools]
2018-03-14 §
20:57 <bd808> Upgrading elasticsearch on tools-elastic-01 (T181531) [tools]