251-300 of 568 results (9ms)
|
2016-05-23
§
|
| 13:28 |
<chasemp> |
'apt-get install hhvm -y --force-yes' across trusty hosts to handle hhvm downgrade |
[tools] |
|
2016-05-20
§
|
| 23:39 |
<bd808> |
Forced puppet run on bastion-02 & bastion-05 to apply fix for T135861 |
[tools] |
| 19:44 |
<chasemp> |
tools-exec-1406 having issues rebooting |
[tools] |
|
2016-05-19
§
|
| 21:07 |
<bd808> |
deployed jobutils 1.13 on bastions; now with '-l release=...' validation! |
[tools] |
| 15:43 |
<YuviPanda> |
rebooting all tools worker instances |
[tools] |
| 13:12 |
<chasemp> |
reboot tools-exec-1220 stuck in state of unresponsivenss |
[tools] |
|
2016-05-13
§
|
| 00:40 |
<YuviPanda> |
cleared all queues that were in error state |
[tools] |
|
2016-05-12
§
|
| 22:59 |
<YuviPanda> |
restart tools-worker-1004 to attempt bringing it back up |
[tools] |
| 22:59 |
<YuviPanda> |
deploy k8s 1.2.4wmf1 on all proxy nodes |
[tools] |
| 22:58 |
<YuviPanda> |
deploy k8s on all worker nodes |
[tools] |
| 22:46 |
<YuviPanda> |
deploy k8s master for 1.2.4wmf1 |
[tools] |
|
2016-05-10
§
|
| 04:25 |
<bd808> |
Added role::package::builder to tools-services-01 |
[tools] |
|
2016-05-09
§
|
| 04:33 |
<YuviPanda> |
reboot tools-worker-1004, lots of ksoftirqd stuckness despite no actual containers running |
[tools] |
|
2016-05-08
§
|
| 07:06 |
<YuviPanda> |
restarted admin tool |
[tools] |
|
2016-05-05
§
|
| 13:11 |
<godog> |
cherry-pick https://gerrit.wikimedia.org/r/#/c/280652/ on puppetmaster |
[tools] |
|
2016-04-28
§
|
| 04:15 |
<YuviPanda> |
delete half of the trusty webservice jobs |
[tools] |
| 04:00 |
<YuviPanda> |
deleted all precise webservice jobs, waiting for webservicemonitor to bring them back up |
[tools] |
|
2016-04-24
§
|
| 12:22 |
<YuviPanda> |
force deleted job 5435259 from pbbot per PeterBowman |
[tools] |
|
2016-04-11
§
|
| 14:20 |
<andrewbogott> |
moving tools-bastion-mtemp to labvirt1009 |
[tools] |
|
2016-04-06
§
|
| 15:20 |
<bd808> |
Removed local hack for T131906 from tools-puppetmaster-01 |
[tools] |
|
2016-03-30
§
|
| 15:50 |
<andrewbogott> |
rebooting tools-proxy-01 in hopes of clearing some bad caches |
[tools] |
|
2016-03-28
§
|
| 20:51 |
<yuvipanda> |
lifted RAM quota from 900Gigs to 1TB?! |
[tools] |
| 20:30 |
<chasemp> |
change perm grant files from create-dbusers for chmod 400 chat chattr +i |
[tools] |
|
2016-03-18
§
|
| 15:47 |
<chasemp> |
had to kill stalkboten as it was logging constant errors filling logs to the tune of hundreds of gigs |
[tools] |
| 15:36 |
<chasemp> |
cleanup huge log collection for broken bot: /srv/project/tools/project/betacommand-dev/tspywiki/irc/logs# rm -fR SpamBotLog.log\.* |
[tools] |
|
2016-03-11
§
|
| 20:57 |
<mutante> |
reverted font changes - puppet runs recovering |
[tools] |
| 20:37 |
<mutante> |
more puppet issues due to font dependencies on trusty, on it |
[tools] |
| 19:39 |
<mutante> |
should a tools-exec server be influenced by font packages on an mw appserver? |
[tools] |
| 19:39 |
<mutante> |
fixed puppet runs on tools-exec (gerrit 276792) |
[tools] |
|
2016-03-02
§
|
| 14:56 |
<chasemp> |
qdel 3956069 and 3758653 for abusing auth |
[tools] |
|
2016-02-28
§
|
| 20:08 |
<bd808> |
Removed unwanted NFS mounts from tools-elastic-01.tools.eqiad.wmflabs |
[tools] |
|
2016-02-26
§
|
| 19:08 |
<bd808> |
Upgraded Elasticsearch on tools-elastic-0[123] to 1.7.5 |
[tools] |
|
2016-02-24
§
|
| 19:46 |
<chasemp> |
runonce deployed for https://gerrit.wikimedia.org/r/#/c/272891/ |
[tools] |
|
2016-02-19
§
|
| 15:58 |
<chasemp> |
rerollout tools nfs shaping pilot for sanity in anticipation of formalization |
[tools] |
| 09:21 |
<_joe_> |
killed cluebot3 instance on tools-exec-1207, writing 20 M/s to the error log |
[tools] |
| 00:50 |
<yuvipanda> |
failover services to services-02 |
[tools] |
|
2016-02-18
§
|
| 22:57 |
<valhallasw`cloud> |
restarted gridengine-master on tools-grid-master, otherwise all webservices will stay down |
[tools] |
| 20:37 |
<yuvipanda> |
failover proxy back to tools-proxy-01 |
[tools] |
| 19:46 |
<chasemp> |
repool labvirt1003 and depool labvirt1004 |
[tools] |
| 18:19 |
<chasemp> |
draining nodes from labvirt1001 |
[tools] |
|
2016-02-16
§
|
| 21:33 |
<chasemp> |
reboot of bastion-1002 |
[tools] |
|
2016-02-12
§
|
| 19:56 |
<chasemp> |
nfs traffic shaping pilot round 2 |
[tools] |
|
2016-02-05
§
|
| 22:01 |
<chasemp> |
throttle some vm nfs write speeds |
[tools] |
|
2016-02-03
§
|
| 03:00 |
<YuviPanda> |
upgraded flannel on all hosts running it |
[tools] |
|
2016-01-29
§
|
| 21:25 |
<YuviPanda> |
restarted image-resize-calc manually, no service.manifest file |
[tools] |
|
2016-01-27
§
|
| 23:07 |
<YuviPanda> |
removed all members of templatetiger, added self instead, removed active shell sessions |
[tools] |
| 20:24 |
<chasemp> |
master stop, truncate accounting log to accounting.01272016, master start |
[tools] |
| 19:34 |
<chasemp> |
master start grid master |
[tools] |
| 19:23 |
<chasemp> |
stopped master |
[tools] |
| 19:11 |
<YuviPanda> |
depooled tools-webgrid-1405 to prep for restart, lots of stuck processes |
[tools] |