601-650 of 943 results (8ms)
2018-03-14 §
20:53 <bd808> Upgrading elasticsearch on tools-elastic-02 (T181531) [tools]
20:51 <bd808> Upgrading elasticsearch on tools-elastic-03 (T181531) [tools]
2018-03-12 §
14:58 <zhuyifei1999_> building, publishing, and deploying misctools 1.31 5f3561e T189430 [tools]
13:31 <arturo> tools-exec-1441 and tools-exec-1442 rebooted fine and are repooled [tools]
13:26 <arturo> depool tools-exec-1441 and tools-exec-1442 for reboots [tools]
2018-03-08 §
14:02 <arturo> T188994 upgrading trusty-tools packages in all the cluster, this includes jobutils, openssh-server and openssh-sftp-server [tools]
2018-03-06 §
15:03 <arturo> drain and reboot tools-worker-1011 [tools]
14:58 <arturo> drain and reboot tools-worker-1010 [tools]
13:21 <arturo> T188994 in some servers there was some race in the dpkg lock between apt-upgrade and puppet. Also, I forgot to use DEBIAN_FRONTEND=noninteractive, so debconf prompts happened and stalled dpkg operations. Already solved, but some puppet alerts were produced [tools]
11:41 <arturo> aborrero@tools-clushmaster-01:~$ clush -w @all "sudo DEBIAN_FRONTEND=noninteractive apt-get autoremove -y" <-- we did in canary servers last week and it went fine. So run in fleet-wide [tools]
11:36 <arturo> (ubuntu) removed linux-image-3.13.0-142-generic and linux-image-3.13.0-137-generic (T188911) [tools]
2018-03-05 §
18:39 <zhuyifei1999_> built and published misctools_1.30_all.deb T167026 T181492 [tools]
2018-02-26 §
21:18 <bstorm_> Deleted tools-static-10 and tools-static-11 now that they are replaced with the much smaller 12 and 13 https://phabricator.wikimedia.org/T182604 [tools]
2018-02-21 §
16:26 <bd808> Rebooting tools-docker-registry-01, NFS mounts are in a bad state [tools]
11:35 <arturo> package upgrades in tools-package-builder-01 tools-prometheus-01 tools-static-10 and tools-redis-1001 [tools]
10:51 <arturo> package upgrades in tools-checker-01 tools-clushmaster-01 and tools-docker-builder-05 [tools]
03:32 <zhuyifei1999_> removed /data/project/.elasticsearch.ini, owned by root and mode 644, leaks the creds of /data/project/strephit/.elasticsearch.ini Might need to cycle it as well... [tools]
2018-02-19 §
18:23 <arturo> upgrade packages of tools-cron-01 from all channels (trusty-wikimedia, trusty-updates and trusty-tools) [tools]
2018-02-16 §
18:21 <arturo> upgrading tools-proxy-01 and tools-paws-master-01, same as others [tools]
2018-02-14 §
13:09 <arturo> the reboot was OK, the server seems working and kubectl sees all the pods running in the deployment (T187315) [tools]
2018-02-09 §
06:15 <bd808> Killed orphan processes owned by iabot, dupdet, and wsexport scattered across the webgrid nodes [tools]
05:07 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1426 [tools]
05:06 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1411 [tools]
05:05 <bd808> Killed 1 orphan php-fcgi process from jembot that were running on tools-webgrid-lighttpd-1409 [tools]
05:02 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1421 and pegging the cpu there [tools]
04:56 <bd808> Rescheduled 30 of the 60 tools running on tools-webgrid-lighttpd-1421 (T186830) [tools]
04:39 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1417 and pegging the cpu there [tools]
2018-01-25 §
23:47 <arturo> fix last deprecation warnings in tools-elastic-03, tools-elastic-02, tools-proxy-01 and tools-proxy-02 by replacing by hand configtimeout with http_configtimeout in /etc/puppet/puppet.conf [tools]
05:25 <arturo> deploying misctools and jobutils 1.29 for T179386 [tools]
2018-01-23 §
15:48 <bd808> Admin clean up; removed Coren, Ryan Lane, and Springle. [tools]
14:17 <chasemp> add me, arturo, chico to sudoers and removed marc [tools]
2018-01-22 §
18:32 <arturo> T181948 T185314 deploying jobutils and misctools v1.28 in the cluster [tools]
11:21 <arturo> puppet in the cluster is mostly fine, except for a couple of deprecation warnings, a conn timeout to services-01 and https://phabricator.wikimedia.org/T181948#3916790 [tools]
2018-01-19 §
12:56 <arturo> the puppet status across the fleet seems good, only minor things like T185314 , T179388 and T179386 [tools]
2018-01-16 §
21:24 <andrewbogott> repooled tools-exec-1420 and tools-webgrid-lighttpd-1417 [tools]
21:14 <andrewbogott> depooling tools-exec-1420 and tools-webgrid-lighttpd-1417 [tools]
20:20 <andrewbogott> depooling tools-webgrid-lighttpd-1412 and tools-exec-1423 for host reboot [tools]
20:00 <andrewbogott> depooled and repooled tools-webgrid-lighttpd-1427 tools-webgrid-lighttpd-1428 tools-exec-1413 tools-exec-1442 for host reboot [tools]
18:50 <andrewbogott> repooling tools-exec-1422 and tools-webgrid-lighttpd-1413 [tools]
18:31 <andrewbogott> depooling tools-exec-1422 and tools-webgrid-lighttpd-1413 for host reboot [tools]
18:26 <andrewbogott> repooling tools-exec-1404 and 1434 for host reboot [tools]
18:06 <andrewbogott> depooling tools-exec-1404 and 1434 for host reboot [tools]
2018-01-11 §
20:33 <andrewbogott> uncordoning tools-worker-1012 and tools-worker-1017 [tools]
20:06 <andrewbogott> cordoning tools-worker-1012 and tools-worker-1017 [tools]
14:46 <chasemp> install metltdown kernel and reboot workers 1011-1016 as jessie pilot [tools]
2018-01-10 §
13:22 <arturo> empty by hand syslog and daemon.log files. They are so big that logrotate won't handle them [tools]
2018-01-09 §
23:08 <yuvipanda> turns out the version of k8s we had wasn't recent enough to support easy upgrades, so destroy entire cluster again and install 1.9.1 [tools]
23:01 <yuvipanda> kill paws master and reboot it [tools]
20:15 <chasemp> disable puppet on proxies and k8s workers [tools]
2018-01-08 §
20:34 <madhuvishy> Restart kube services and uncordon tools-worker-1001 [tools]