6201-6250 of 7884 results (28ms)
2014-08-27 §
22:46 <hashar> blackholed an IP address on deployment-cache-text02 and deployment-cache-mobile03 , it was causing hundred of requests per seconds and overloaded the beta cluster. Use route -n to find the IP [releng]
22:37 <hashar> restarting udp2log-mw on deployment-bastion. It keeps crashing since fiarly recently [releng]
22:26 <bd808> when restarting varnish on deployment-cache-text02, don't forget that there are 2 varnish services (varnish and varnish-frontend) [releng]
22:19 <bd808> restarted varnish (again) on deployment-cache-text02 [releng]
22:10 <bd808> restarted varnish on deployment-cache-text02 [releng]
16:22 <bd808> killing `apt-get update` process running on deployment-bastion since Jun13 [releng]
14:59 <bd808> Resolved puppet git merge conflict on deployment-salt [releng]
14:49 <bd808> Moved hhvm core dumps to /data/project/hhvm-cores [releng]
14:42 <bd808> Root dirve full on deployment-mediawiki02; hhvm core files are the culprit [releng]
2014-08-25 §
23:47 <ori> stopping hhvm/apache on deployment-mediawiki02 to replace debug build of hhvm with release build [releng]
21:44 <bd808> Deployed scap 116027f (Make sync-common update l10n cdb files by default) [releng]
18:30 <ori> deployment-mediawiki02: cleared /tmp; running puppet [releng]
15:05 <hashar> mediawiki02 rm /tmp/hhvm*.core . Filled as {{bug|69979}} [releng]
15:01 <hashar> mediawiki02 rm /tmp/mw-cache-master/conf* [releng]
15:01 <hashar> mediawiki02 has mw conf caches under /tmp/mw-cache-master/ and since that partition is filled up, that ends up with conf caches being null file [releng]
15:00 <hashar> mediawiki02 rm /var/log/upstart/hhvm* [releng]
14:53 <hashar> mediawiki02 : removed /var/lib/puppet/state/agent_catalog_run.lock [releng]
14:46 <hashar> restarting udp2log-mw service on -bastion. It is stalled for some reason [releng]
14:42 <hashar> on mediawiki02 , clearing out some /var/log/upstart/hhvm.* log files see {{bug|69976}} [releng]
14:34 <hashar> mediawiki02 / partition is 100% full [releng]
2014-08-22 §
20:21 <hashar> udp2log are back in /data/project/logs . The udp2log-mw service went stall for some reason. [releng]
20:08 <ori> ran 'git pull' on deployment-salt:/srv/var-lib/git/operations/puppet [releng]
19:59 <hashar> restarting udp2log-mw service on deployment-bastion [releng]
19:59 <hashar> bits yielding 503 [releng]
00:41 <bd808> cherry-picked scap change https://gerrit.wikimedia.org/r/#/c/155677/ for testing [releng]
2014-08-21 §
21:49 <bd808> Trebuchet happier after all the salt-minion restarts; still have deleted hosts showing in the expected minion list for scap deploys [releng]
21:01 <twentyafterfour> Started salt-minion on deployment-redis01 [releng]
21:01 <bd808> Started salt-minon on deployment-upload [releng]
21:00 <bd808> Started salt-minon on deployment-fluoride [releng]
21:00 <bd808> Started salt-minon on deployment-db1 [releng]
20:59 <bd808> Started salt-minon on deployment-elastic01 [releng]
20:59 <twentyafterfour> Started salt-minion on deployment-eventlogging02 [releng]
20:58 <bd808> Started salt-minon on deployment-elastic02 [releng]
20:58 <bd808> Started salt-minon on deployment-elastic03 [releng]
20:57 <bd808> Started salt-minon on deployment-elastic04 [releng]
20:57 <bd808> Started salt-minon on deployment-analytics01 [releng]
20:55 <bd808> Started salt-minon on deployment-cache-upload02 [releng]
20:54 <bd808> Started salt-minon on deployment-memc04 [releng]
20:54 <bd808> Started salt-minon on deployment-parsoid04 [releng]
20:49 <bd808> Started salt-minon on deployment-memc05 [releng]
20:48 <bd808> Started salt-minon on deployment-db2 [releng]
20:48 <twentyafterfour> Started salt-minion on deployment-cache-text02 [releng]
20:47 <twentyafterfour> Started salt-minion on deployment-memc03 [releng]
20:47 <bd808> Started salt-minon on deployment-cxserver01 [releng]
20:12 <bd808> List of broken salt minions can be obtained with `sudo salt-run manage.down` on deployment-salt [releng]
19:55 <bd808> Fixed salt on deployment-memc02 [releng]
19:52 <bd808> Salt minions are broken all over beta. Hung grain-ensure calls, hung test.ping calls, downed minions [releng]
19:50 <bd808> Killed dozens of grain-ensure calls and started salt-minion on deployment-cache-mobile03 [releng]
19:47 <bd808> Killed hung salt-call and started salt-minion on deployment-cache-bits01 [releng]
19:28 <bd808> Deployed cherry-pick of Iea7217a for scap [releng]