|
2014-08-25
§
|
| 15:05 |
<hashar> |
mediawiki02 rm /tmp/hhvm*.core . Filled as {{bug|69979}} |
[releng] |
| 15:01 |
<hashar> |
mediawiki02 rm /tmp/mw-cache-master/conf* |
[releng] |
| 15:01 |
<hashar> |
mediawiki02 has mw conf caches under /tmp/mw-cache-master/ and since that partition is filled up, that ends up with conf caches being null file |
[releng] |
| 15:00 |
<hashar> |
mediawiki02 rm /var/log/upstart/hhvm* |
[releng] |
| 14:53 |
<hashar> |
mediawiki02 : removed /var/lib/puppet/state/agent_catalog_run.lock |
[releng] |
| 14:46 |
<hashar> |
restarting udp2log-mw service on -bastion. It is stalled for some reason |
[releng] |
| 14:42 |
<hashar> |
on mediawiki02 , clearing out some /var/log/upstart/hhvm.* log files see {{bug|69976}} |
[releng] |
| 14:34 |
<hashar> |
mediawiki02 / partition is 100% full |
[releng] |
|
2014-08-21
§
|
| 21:49 |
<bd808> |
Trebuchet happier after all the salt-minion restarts; still have deleted hosts showing in the expected minion list for scap deploys |
[releng] |
| 21:01 |
<twentyafterfour> |
Started salt-minion on deployment-redis01 |
[releng] |
| 21:01 |
<bd808> |
Started salt-minon on deployment-upload |
[releng] |
| 21:00 |
<bd808> |
Started salt-minon on deployment-fluoride |
[releng] |
| 21:00 |
<bd808> |
Started salt-minon on deployment-db1 |
[releng] |
| 20:59 |
<bd808> |
Started salt-minon on deployment-elastic01 |
[releng] |
| 20:59 |
<twentyafterfour> |
Started salt-minion on deployment-eventlogging02 |
[releng] |
| 20:58 |
<bd808> |
Started salt-minon on deployment-elastic02 |
[releng] |
| 20:58 |
<bd808> |
Started salt-minon on deployment-elastic03 |
[releng] |
| 20:57 |
<bd808> |
Started salt-minon on deployment-elastic04 |
[releng] |
| 20:57 |
<bd808> |
Started salt-minon on deployment-analytics01 |
[releng] |
| 20:55 |
<bd808> |
Started salt-minon on deployment-cache-upload02 |
[releng] |
| 20:54 |
<bd808> |
Started salt-minon on deployment-memc04 |
[releng] |
| 20:54 |
<bd808> |
Started salt-minon on deployment-parsoid04 |
[releng] |
| 20:49 |
<bd808> |
Started salt-minon on deployment-memc05 |
[releng] |
| 20:48 |
<bd808> |
Started salt-minon on deployment-db2 |
[releng] |
| 20:48 |
<twentyafterfour> |
Started salt-minion on deployment-cache-text02 |
[releng] |
| 20:47 |
<twentyafterfour> |
Started salt-minion on deployment-memc03 |
[releng] |
| 20:47 |
<bd808> |
Started salt-minon on deployment-cxserver01 |
[releng] |
| 20:12 |
<bd808> |
List of broken salt minions can be obtained with `sudo salt-run manage.down` on deployment-salt |
[releng] |
| 19:55 |
<bd808> |
Fixed salt on deployment-memc02 |
[releng] |
| 19:52 |
<bd808> |
Salt minions are broken all over beta. Hung grain-ensure calls, hung test.ping calls, downed minions |
[releng] |
| 19:50 |
<bd808> |
Killed dozens of grain-ensure calls and started salt-minion on deployment-cache-mobile03 |
[releng] |
| 19:47 |
<bd808> |
Killed hung salt-call and started salt-minion on deployment-cache-bits01 |
[releng] |
| 19:28 |
<bd808> |
Deployed cherry-pick of Iea7217a for scap |
[releng] |
| 19:27 |
<bd808> |
Restarted salt-minion on deployment-jobrunner01 & deployment-videoscaler01 |
[releng] |
| 19:27 |
<bd808> |
Killed rogue salt-master process on deployment-bastion |
[releng] |
| 19:26 |
<bd808> |
Deleted salt keys for retired apache0[12] minions |
[releng] |
| 00:13 |
<bd808> |
Upgraded elasticsearch to 1.3.2 on deployment-logstash1 |
[releng] |
|
2014-08-19
§
|
| 16:11 |
<hashar> |
deleted /usr/local/apache/common-local symlink, made it a directory and retriggered https://integration.wikimedia.org/ci/job/beta-scap-eqiad/17887/console |
[releng] |
| 16:03 |
<bd808> |
Removed local changes to /usr/local/apache/conf/wmflabs-logging.conf on deployment-mediawiki02; logs back to nfs share |
[releng] |
| 15:52 |
<bd808> |
Changed apache logging level from debug to notice on deployment-mediawiki02 in /usr/local/apache/conf/wmflabs-logging.conf |
[releng] |
| 15:47 |
<bd808> |
Changed apache logging level from debug to warn on deployment-mediawiki02 |
[releng] |
| 15:44 |
<bd808> |
/var full on deployment-mediawiki02; deleting 572M /var/log/apache2/debug.log.1 |
[releng] |
| 15:03 |
<hashar> |
Killed some stalled scap / rsync process on deployment-bastion that were preventing https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ from acquiring the lock. |
[releng] |
| 14:17 |
<hashar> |
huge rsync in progress on bastion |
[releng] |
| 14:00 |
<hashar> |
On bastion reverted the symlink on bastion and manually created directory /usr/local/apache/common-local |
[releng] |