2851-2900 of 7896 results (20ms)
2016-08-31 §
23:19 <bd808> Deleted /data/scratch on integration-slave-trusty-1011 to fix puppet [releng]
23:15 <bd808> Deleted /data/scratch on integration-slave-precise-1012 to fix puppet [releng]
23:11 <bd808> Deleted /data on integration-slave-precise-1011 to fix puppet [releng]
23:08 <bd808> Deleted /data on integration-slave-jessie-1001 to fix puppet [releng]
23:04 <bd808> Deleted empty /data, /data/project, and /data/scratch on integration-puppetmaster to fix puppet [releng]
22:59 <bd808> Deleted empty /data, /data/project, and /data/scratch on integration-publisher to fix puppet [releng]
01:44 <Krinkle> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/307670 [releng]
2016-08-30 §
23:31 <yuvipanda> cherry-picking https://gerrit.wikimedia.org/r/#/c/307656/ fixed puppet on the elasticsearch machines! [releng]
23:20 <Krenair> removed 'project_id' key from deployment-restbase02's metadata to fix compatibility with the new labsprojectfrommetadata code [releng]
22:29 <yuvipanda> in lieu of blood sacrifice, restart puppetmaster on deployment-pupetmaster [releng]
21:43 <yuvipanda> use clush to fix puppet.conf of all clients, realize also accidentally set a client's puppet.conf for the server, recover server's old conf file from a cat in shell history, restore, breathe sigh of relief [releng]
21:37 <yuvipanda> sudo takes like 15s each time, is there no god? [releng]
21:36 <yuvipanda> managed to get vim into a state where I can not quit it, probably recording a macro. I hate computers [releng]
21:16 <yuvipanda> deployment-pdf01 fixed manually [releng]
21:15 <yuvipanda> deployment-pdf02 has proper ssl certs mysteriously without me doing anything [releng]
21:06 <yuvipanda> moved deployment-db[12], deployment-stream to not use role::puppet::self, attempting to semi-automate rest [releng]
20:52 <yuvipanda> cherry-picked appropriate patch on deployment-puppetmaster for T120159, did https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep/host/deployment-puppetmaster&oldid=818847 to make sure the puppetmaster allows connections from elsewhere [releng]
19:48 <legoktm> deploying https://gerrit.wikimedia.org/r/306710 [releng]
19:13 <bd808> Fixed puppet runs on deployment-sca0[12] with cherry-pick of https://gerrit.wikimedia.org/r/#/c/307561 [releng]
18:57 <bd808> Duplicate declaration: File[/srv/deployment] is already declared in file /etc/puppet/modules/contint/manifests/deployment_dir.pp:14; cannot redeclare at /etc/puppet/modules/service/manifests/deploy/common.pp:12 on node deployment-sca01.deployment-prep.eqiad.wmflabs [releng]
18:40 <bd808> Puppet busted on deployment-aqs01 -- Could not find data item analytics_hadoop_hosts in any Hiera data file and no default supplied at /etc/puppet/manifests/role/aqs.pp:46 [releng]
18:09 <yuvipanda> reboot deployment-kafka03 seems to be stuck [releng]
12:59 <hashar> beta: revert master branch to origin. Ran scap and enabled again beta-code-update-eqiad job. [releng]
12:55 <hashar> Running scap on beta cluster via https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/117786/console T143889 [releng]
12:53 <hashar> Cherry picking https://gerrit.wikimedia.org/r/#/c/307501/ on beta cluster for T143889 [releng]
12:51 <hashar> disabling https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/ to cherry pick a revert patch [releng]
2016-08-29 §
07:56 <hashar> hard rebooting integration-slave-trusty-1012 via horizon and restarting puppet manually [releng]
07:50 <hashar> integration-slave-trusty-1013 puppet.conf certname was set to 'undef' breaking puppet [releng]
2016-08-27 §
20:51 <hashar> integration: tweak sudo policy for jenkins-deploy running cowbuilder: env_keep+=DEB_BUILD_OPTIONS [releng]
20:24 <hashar> Manually installing jenkins-debian-glue 0.17.0 on integration-slave-jessie-1004 and integration-slave-jessie-1005 ( T142891 ) . That is to support PBUILDER_USENETWORK T141114 [releng]
20:05 <hashar> Jenkins added global env variable BUILD_TIMEOUT set to 30 for T144094 [releng]
2016-08-26 §
22:29 <legoktm> deploying https://gerrit.wikimedia.org/r/307025 [releng]
08:15 <Amir1> restart uwsgi-ores and celery-ores-worker in deployment-sca03 (T143567) [releng]
08:10 <hashar> beta-scap-eqiad job is back in operation. Was blocked on logstash not being reachable. T143982 [releng]
08:10 <hashar> deployment-logstash2 is back after a hard reboot. T143982 [releng]
08:07 <hashar> rebooting deployment-logstash02 via Horizon. Kernel hang apparently T143982 [releng]
08:00 <hashar> beta-scap-eqiad failing investigating [releng]
07:54 <Amir1> cherry-picked 306839/1 into deployment-puppetmaster [releng]
00:28 <twentyafterfour> restarted puppetmaster service on deployment-puppetmaster [releng]
2016-08-25 §
23:15 <Amir1> cherry-picked 306839/1 into puppetmaster [releng]
20:10 <hashar> Delete integration-slave-trusty-1023 with label AndroidEmulator. The Android job has been migrated to a new Jessie based instance via T138506 [releng]
19:05 <hashar> hard rebooting integration-raita via Horizon [releng]
16:04 <hashar> fixing puppet.conf on integration-slave-trusty-1013 it mysteriously considered itself as the puppetmaster [releng]
16:02 <hashar> integration restarted puppetmaster service [releng]
08:28 <hashar> beta update database fixed [releng]
08:28 <hashar> beta cluster update database failed due to: "Your composer.lock file is up to date with current dependencies!" Probably a race condition with ongoing scap. [releng]
2016-08-24 §
15:14 <halfak> deploying ores d00171 [releng]
09:50 <hashar> deployment-redis02 fixed AOF file /srv/redis/deployment-redis02-6379.aof and restarted the redis instance should fix T143655 and might help T142600 [releng]
09:43 <hashar> T143655 stopping redis 6379 on deployment-redis02 : initctl stop redis-instance-tcp_6379 [releng]
09:38 <hashar> deployment-redis02 initctl stop redis-instance-tcp_6379 && initctl start redis-instance-tcp_6379 | That did not fix it magically though T143655 [releng]