2951-3000 of 4803 results (14ms)
2014-09-30 §
20:02 <bd808> Started building deployment-rsync02 to replace deployment-rsync01 [releng]
19:59 <bd808|LUNCH> /srv partition on deployment-rsync01 full again. We need a new rsync server with more space [releng]
17:44 <bd808> Updated scap to 064425b (Remove restart-nutcracker and restart-twemproxy scripts) [releng]
16:08 <bd808> Occasional memecached-serious errors in beta for something trying to connect to the default memcached port (11211) rather than the nutcracker port (11212). [releng]
15:58 <bd808> scap happy again after fixing rogue group/user on rsync01 \\o/ Not sure why they were created but likely an ldap hiccup during a puppet run [releng]
15:56 <bd808> removed local group/user mwdeploy on deployment-rsync01 [releng]
15:54 <bd808> Local mwdeploy (gid=996) shadowing ldap group gid=603(mwdeploy) on deployment-rsync01 [releng]
15:49 <bd808> apt-get dist-upgrade fixed hhvm on deployment-mediawiki03 [releng]
15:45 <hashar> Updating our Jenkins job builder fork 686265a..ee80dbc (no job changed) [releng]
15:44 <bd808> scap failing in beta due to "Permission denied (publickey)" talking to deployment-rsync01.eqiad.wmflabs [releng]
15:39 <bd808> hhvm not starting after puppet run on deployment-mediawiki03. Investigating. [releng]
15:36 <bd808> enabling puppet and forcing run on deployment-mediawiki03 [releng]
15:36 <bd808> enabling puppet and forcing run on deployment-mediawiki03 [releng]
15:34 <bd808> enabling puppet and forcing run on deployment-mediawiki02 [releng]
15:34 <bd808> enabling puppet and forcing run on deployment-mediawiki02 [releng]
15:29 <bd808> puppet showed no changes on mediawiki01‽ [releng]
15:28 <bd808> enabling puppet and forcing run on deployment-mediawiki01 [releng]
15:27 <bd808> enabling puppet and forcing run on deployment-mediawiki01 [releng]
15:13 <bd808> Fixed logstash by installing http://packages.elasticsearch.org/logstash/1.4/debian/pool/main/l/logstash-contrib/logstash-contrib_1.4.2-1-efd53ef_all.deb [releng]
15:02 <bd808> Logstash doesn't bundle the prune filter by default any more -- http://logstash.net/docs/1.4.2/filters/prune [releng]
14:59 <bd808> Logstash rules need to be adjusted for latest upstream version: "Couldn't find any filter plugin named 'prune'" [releng]
12:37 <hashar> Fixed some file permissions under deployment-bastion:/srv/mediawiki-staging/php-master/vendor/.git some files belonged to root instead of mwdeploy [releng]
00:34 <bd808> Updated kibana to latest upstream head 8653aba [releng]
2014-09-29 §
22:45 <Reedy> re-enabled beta-scap-eqiad [releng]
21:34 <Reedy> disabled "beta-scap-eqiad" until things are fixed [releng]
21:24 <Reedy> deleted l10n cache on deployment-rsync01 to attempt to run sync-common manually [releng]
21:22 <Reedy> deployment-rsync01 hard drive is far too small [releng]
17:57 <cscott> updated OCG to version 89d8f29a24295b05d0643abe976fea83b56575c9 [releng]
14:22 <hashar> apt-get upgrade and reboot of all integration-slaveXX instances [releng]
14:07 <hashar> updated puppetmaster labs/private on both integration and beta cluster projects ( a41fcdd..84f0906 ) [releng]
08:57 <hashar> rebased puppetmaster [releng]
06:58 <ori> Configured Beta cluster to use redis for session storage [releng]
06:57 <ori> Created deployment-redis02 and converted it to use local puppet & salt masters [releng]
05:23 <ori> Created deployment-redis01 and converted it to use local puppet & salt masters [releng]
2014-09-28 §
14:38 <andrewbogott> cherry-picked https://gerrit.wikimedia.org/r/#/c/163464/ onto deployment-salt to fix a puppet compile failure. [releng]
14:38 <andrewbogott> edited and re-cherry-picked roan's citoid patch into beta because the previous version was breaking puppet [releng]
2014-09-26 §
22:16 <bd808> Deleted deployment-mediawiki04 (i-000005ba.eqiad.wmflabs) and removed from salt and trebuchet [releng]
07:50 <hashar> Pooled back integration-slave1006 , was removed because of {{bug|71314}} [releng]
07:41 <hashar> Updated our Jenkins Job Builder fork 2d74b16..686265a [releng]
06:34 <cscott> updated OCG to version f3a6c1cbba118d4a5e1aa019937dc50159fc823d [releng]
2014-09-25 §
23:35 <bd808> Done messing with puppet repo. Replaced 2 local commits with proper gerrit cherry picks. Removed a cherry-pick that had been rearranged and merged. Removed a cherry-pick that had been abandoned in gerrit. [releng]
23:10 <bd808> removed cherry-pick of abandoned https://gerrit.wikimedia.org/r/#/c/156223/; if beta wikis stop working this would be a likely culprit [releng]
22:48 <RoanKattouw> Fixed permissions of deployment-bastion:/srv/deployment/mathoid/mathoid/.git/deploy (needed g+w) [releng]
22:36 <bd808> Trying to reduce the number of untracked changes in puppet repo. Expect some short term breakage. [releng]
22:21 <bd808> cleaned up puppet repo with `git rebase origin/production; git submodule update --init --recursive` [releng]
22:18 <bd808> puppet repo on deployment-salt out of whack. I will try to fix. [releng]
11:36 <_joe_> updated hhvm to fix most bugs, also cherry-picked https://gerrit.wikimedia.org/r/#/c/162839/ [releng]
08:15 <hashar> beta: puppetmaster rebased [releng]
08:10 <hashar> beta: dropped a patch that reverted OCG LVS configuration ( https://gerrit.wikimedia.org/r/#/c/146860/ ), it has been fixed by https://gerrit.wikimedia.org/r/#/c/148371/ [releng]
08:04 <hashar> attempting to rebase beta cluster puppet master. Currently at 74036376 [releng]