|
2014-09-30
§
|
| 20:35 |
<bd808> |
Initial puppet run with role::beta::rsync_slave applied on rsync02 failed spectacularly in /Stage[main]/Mediawiki::Scap/Exec[fetch_mediawiki] stage |
[releng] |
| 20:26 |
<bd808> |
Converted deployment-rsync02 to use local puppet & salt masters |
[releng] |
| 20:02 |
<bd808> |
Started building deployment-rsync02 to replace deployment-rsync01 |
[releng] |
| 19:59 |
<bd808|LUNCH> |
/srv partition on deployment-rsync01 full again. We need a new rsync server with more space |
[releng] |
| 17:44 |
<bd808> |
Updated scap to 064425b (Remove restart-nutcracker and restart-twemproxy scripts) |
[releng] |
| 16:08 |
<bd808> |
Occasional memecached-serious errors in beta for something trying to connect to the default memcached port (11211) rather than the nutcracker port (11212). |
[releng] |
| 15:58 |
<bd808> |
scap happy again after fixing rogue group/user on rsync01 \\o/ Not sure why they were created but likely an ldap hiccup during a puppet run |
[releng] |
| 15:56 |
<bd808> |
removed local group/user mwdeploy on deployment-rsync01 |
[releng] |
| 15:54 |
<bd808> |
Local mwdeploy (gid=996) shadowing ldap group gid=603(mwdeploy) on deployment-rsync01 |
[releng] |
| 15:49 |
<bd808> |
apt-get dist-upgrade fixed hhvm on deployment-mediawiki03 |
[releng] |
| 15:45 |
<hashar> |
Updating our Jenkins job builder fork 686265a..ee80dbc (no job changed) |
[releng] |
| 15:44 |
<bd808> |
scap failing in beta due to "Permission denied (publickey)" talking to deployment-rsync01.eqiad.wmflabs |
[releng] |
| 15:39 |
<bd808> |
hhvm not starting after puppet run on deployment-mediawiki03. Investigating. |
[releng] |
| 15:36 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki03 |
[releng] |
| 15:36 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki03 |
[releng] |
| 15:34 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki02 |
[releng] |
| 15:34 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki02 |
[releng] |
| 15:29 |
<bd808> |
puppet showed no changes on mediawiki01‽ |
[releng] |
| 15:28 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki01 |
[releng] |
| 15:27 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki01 |
[releng] |
| 15:13 |
<bd808> |
Fixed logstash by installing http://packages.elasticsearch.org/logstash/1.4/debian/pool/main/l/logstash-contrib/logstash-contrib_1.4.2-1-efd53ef_all.deb |
[releng] |
| 15:02 |
<bd808> |
Logstash doesn't bundle the prune filter by default any more -- http://logstash.net/docs/1.4.2/filters/prune |
[releng] |
| 14:59 |
<bd808> |
Logstash rules need to be adjusted for latest upstream version: "Couldn't find any filter plugin named 'prune'" |
[releng] |
| 12:37 |
<hashar> |
Fixed some file permissions under deployment-bastion:/srv/mediawiki-staging/php-master/vendor/.git some files belonged to root instead of mwdeploy |
[releng] |
| 00:34 |
<bd808> |
Updated kibana to latest upstream head 8653aba |
[releng] |
|
2014-09-29
§
|
| 22:45 |
<Reedy> |
re-enabled beta-scap-eqiad |
[releng] |
| 21:34 |
<Reedy> |
disabled "beta-scap-eqiad" until things are fixed |
[releng] |
| 21:24 |
<Reedy> |
deleted l10n cache on deployment-rsync01 to attempt to run sync-common manually |
[releng] |
| 21:22 |
<Reedy> |
deployment-rsync01 hard drive is far too small |
[releng] |
| 17:57 |
<cscott> |
updated OCG to version 89d8f29a24295b05d0643abe976fea83b56575c9 |
[releng] |
| 14:22 |
<hashar> |
apt-get upgrade and reboot of all integration-slaveXX instances |
[releng] |
| 14:07 |
<hashar> |
updated puppetmaster labs/private on both integration and beta cluster projects ( a41fcdd..84f0906 ) |
[releng] |
| 08:57 |
<hashar> |
rebased puppetmaster |
[releng] |
| 06:58 |
<ori> |
Configured Beta cluster to use redis for session storage |
[releng] |
| 06:57 |
<ori> |
Created deployment-redis02 and converted it to use local puppet & salt masters |
[releng] |
| 05:23 |
<ori> |
Created deployment-redis01 and converted it to use local puppet & salt masters |
[releng] |
|
2014-09-25
§
|
| 23:35 |
<bd808> |
Done messing with puppet repo. Replaced 2 local commits with proper gerrit cherry picks. Removed a cherry-pick that had been rearranged and merged. Removed a cherry-pick that had been abandoned in gerrit. |
[releng] |
| 23:10 |
<bd808> |
removed cherry-pick of abandoned https://gerrit.wikimedia.org/r/#/c/156223/; if beta wikis stop working this would be a likely culprit |
[releng] |
| 22:48 |
<RoanKattouw> |
Fixed permissions of deployment-bastion:/srv/deployment/mathoid/mathoid/.git/deploy (needed g+w) |
[releng] |
| 22:36 |
<bd808> |
Trying to reduce the number of untracked changes in puppet repo. Expect some short term breakage. |
[releng] |
| 22:21 |
<bd808> |
cleaned up puppet repo with `git rebase origin/production; git submodule update --init --recursive` |
[releng] |
| 22:18 |
<bd808> |
puppet repo on deployment-salt out of whack. I will try to fix. |
[releng] |
| 11:36 |
<_joe_> |
updated hhvm to fix most bugs, also cherry-picked https://gerrit.wikimedia.org/r/#/c/162839/ |
[releng] |
| 08:15 |
<hashar> |
beta: puppetmaster rebased |
[releng] |