|
2014-08-21
§
|
| 20:12 |
<bd808> |
List of broken salt minions can be obtained with `sudo salt-run manage.down` on deployment-salt |
[releng] |
| 19:55 |
<bd808> |
Fixed salt on deployment-memc02 |
[releng] |
| 19:52 |
<bd808> |
Salt minions are broken all over beta. Hung grain-ensure calls, hung test.ping calls, downed minions |
[releng] |
| 19:50 |
<bd808> |
Killed dozens of grain-ensure calls and started salt-minion on deployment-cache-mobile03 |
[releng] |
| 19:47 |
<bd808> |
Killed hung salt-call and started salt-minion on deployment-cache-bits01 |
[releng] |
| 19:28 |
<bd808> |
Deployed cherry-pick of Iea7217a for scap |
[releng] |
| 19:27 |
<bd808> |
Restarted salt-minion on deployment-jobrunner01 & deployment-videoscaler01 |
[releng] |
| 19:27 |
<bd808> |
Killed rogue salt-master process on deployment-bastion |
[releng] |
| 19:26 |
<bd808> |
Deleted salt keys for retired apache0[12] minions |
[releng] |
| 00:13 |
<bd808> |
Upgraded elasticsearch to 1.3.2 on deployment-logstash1 |
[releng] |
|
2014-08-19
§
|
| 16:11 |
<hashar> |
deleted /usr/local/apache/common-local symlink, made it a directory and retriggered https://integration.wikimedia.org/ci/job/beta-scap-eqiad/17887/console |
[releng] |
| 16:03 |
<bd808> |
Removed local changes to /usr/local/apache/conf/wmflabs-logging.conf on deployment-mediawiki02; logs back to nfs share |
[releng] |
| 15:52 |
<bd808> |
Changed apache logging level from debug to notice on deployment-mediawiki02 in /usr/local/apache/conf/wmflabs-logging.conf |
[releng] |
| 15:47 |
<bd808> |
Changed apache logging level from debug to warn on deployment-mediawiki02 |
[releng] |
| 15:44 |
<bd808> |
/var full on deployment-mediawiki02; deleting 572M /var/log/apache2/debug.log.1 |
[releng] |
| 15:03 |
<hashar> |
Killed some stalled scap / rsync process on deployment-bastion that were preventing https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ from acquiring the lock. |
[releng] |
| 14:17 |
<hashar> |
huge rsync in progress on bastion |
[releng] |
| 14:00 |
<hashar> |
On bastion reverted the symlink on bastion and manually created directory /usr/local/apache/common-local |
[releng] |
| 13:55 |
<hashar_> |
On bastion, deleting /usr/local/apache/common-local and symlink it to /srv/common-local |
[releng] |
|
2014-08-18
§
|
| 22:22 |
<^d> |
dropped apache01/02 instances, unused and need the resources |
[releng] |
| 18:23 |
<manybubbles> |
finished upgrading elasticsearch in beta - everything seems ok so far |
[releng] |
| 18:15 |
<bd808> |
Restarted salt-minion on deployment-mediawiki01 & deployment-rsync01 |
[releng] |
| 18:15 |
<bd808> |
Ran `sudo pkill python` on deployment-rsync01 to kill hundreds of grain-ensure processes |
[releng] |
| 18:12 |
<bd808> |
Ran `sudo pkill python` on deployment-mediawiki01 to kill hundreds of grain-ensure processes |
[releng] |
| 18:10 |
<manybubbles> |
finally restarting beta's elasticsearch servers now that they have new jars |
[releng] |
| 17:56 |
<bd808> |
Manually ran trebuchet fetches on deployment-elastic0* |
[releng] |
| 17:49 |
<bd808> |
Forcing puppet run on deployment-elastic01 |
[releng] |
| 17:47 |
<godog> |
upgraded hhvm on mediawiki02 to 3.3-dev+20140728+wmf5 |
[releng] |
| 17:44 |
<bd808> |
Trying to restart minions again with `salt '*' -b 1 service.restart salt-minion` |
[releng] |
| 17:39 |
<bd808> |
Restarting minions via `salt '*' service.restart salt-minion` |
[releng] |
| 17:38 |
<bd808> |
Restarted salt-master service on deployment-salt |
[releng] |
| 17:19 |
<bd808> |
16:37 Restarted Apache and HHVM on deployment-mediawiki02 to pick up removal of /etc/php5/conf.d/mail.ini (logged in prod SAL by mistake) |
[releng] |
| 16:59 |
<manybubbles|lunc> |
upgrading Elasticsearch in beta to 1.3.2 |
[releng] |
| 16:11 |
<bd808> |
Manually applied https://gerrit.wikimedia.org/r/#/c/141287/12/templates/mail/exim4.minimal.erb on deployment-mediawiki02 and restarted exim4 service |
[releng] |
| 15:28 |
<bd808> |
Puppet failing for deployment-mathoid due to duplicate definition error in trebuchet config |
[releng] |
| 15:15 |
<bd808> |
Reinstated puppet patch to depool deployment-mediawiki01 and forced puppet run on all deployment-cache-* hosts |
[releng] |
| 15:04 |
<bd808> |
Puppet run failing on deployment-mediawiki01 (apache won't start); Puppet disabled on deployment-mediawiki02 ('reason not specified') Probably needs to wait until Giuseppe is back from vacation for fixing. |
[releng] |
| 15:00 |
<bd808> |
Rebooting deployment-eventlogging02 via wikitech; console filling with OOM killer messages and puppet runs failing with "Cannot allocate memory - fork(2)" |
[releng] |
| 14:29 |
<bd808> |
Forced puppet run on deployment-cache-upload02 |
[releng] |
| 14:27 |
<bd808> |
Forced puppet run on deployment-cache-text02 |
[releng] |
| 14:24 |
<bd808> |
Forced puppet run on deployment-cache-mobile03 |
[releng] |
| 14:20 |
<bd808> |
Forced puppet run on deployment-cache-bits01 |
[releng] |