6401-6450 of 10000 results (47ms)
2016-08-26 §
08:15 <Amir1> restart uwsgi-ores and celery-ores-worker in deployment-sca03 (T143567) [releng]
08:10 <hashar> beta-scap-eqiad job is back in operation. Was blocked on logstash not being reachable. T143982 [releng]
08:10 <hashar> deployment-logstash2 is back after a hard reboot. T143982 [releng]
08:07 <hashar> rebooting deployment-logstash02 via Horizon. Kernel hang apparently T143982 [releng]
08:00 <hashar> beta-scap-eqiad failing investigating [releng]
07:54 <Amir1> cherry-picked 306839/1 into deployment-puppetmaster [releng]
00:28 <twentyafterfour> restarted puppetmaster service on deployment-puppetmaster [releng]
2016-08-25 §
23:15 <Amir1> cherry-picked 306839/1 into puppetmaster [releng]
20:10 <hashar> Delete integration-slave-trusty-1023 with label AndroidEmulator. The Android job has been migrated to a new Jessie based instance via T138506 [releng]
19:05 <hashar> hard rebooting integration-raita via Horizon [releng]
16:04 <hashar> fixing puppet.conf on integration-slave-trusty-1013 it mysteriously considered itself as the puppetmaster [releng]
16:02 <hashar> integration restarted puppetmaster service [releng]
08:28 <hashar> beta update database fixed [releng]
08:28 <hashar> beta cluster update database failed due to: "Your composer.lock file is up to date with current dependencies!" Probably a race condition with ongoing scap. [releng]
2016-08-24 §
15:14 <halfak> deploying ores d00171 [releng]
09:50 <hashar> deployment-redis02 fixed AOF file /srv/redis/deployment-redis02-6379.aof and restarted the redis instance should fix T143655 and might help T142600 [releng]
09:43 <hashar> T143655 stopping redis 6379 on deployment-redis02 : initctl stop redis-instance-tcp_6379 [releng]
09:38 <hashar> deployment-redis02 initctl stop redis-instance-tcp_6379 && initctl start redis-instance-tcp_6379 | That did not fix it magically though T143655 [releng]
2016-08-23 §
18:21 <legoktm> deploying https://gerrit.wikimedia.org/r/306257 [releng]
16:38 <bd808> Fixed ops/puppet sync by removing stale cherry-pick of https://gerrit.wikimedia.org/r/#/c/305996/ [releng]
08:22 <hashar> running puppet on integration-slave-trusty-1014 [releng]
08:18 <hashar> reboot integration-slave-trusty-1014 [releng]
08:16 <hashar> disabled/enabled Jenkins Gearman client to remove deadlock with Throttle plugin [releng]
2016-08-22 §
23:40 <legoktm> updating slave_scripts on all slaves [releng]
2016-08-19 §
00:39 <Krenair> deployment-fluorine is now deployment-fluorine02 running jessie with the old precise packages shoehorned in [releng]
2016-08-18 §
22:03 <bd808> deployment-fluorine02: Hack 'datasets:x:10003:997::/home/datasets:/bin/bash' into /etc/passwd for T117028 [releng]
20:30 <MaxSem> Restarted hhvm on appservers for wikidiff2 upgrades [releng]
19:03 <MaxSem> Upgrading hhvm-wikidiff2 in beta cluster [releng]
16:53 <legoktm> deploying https://gerrit.wikimedia.org/r/#/c/305532/ [releng]
2016-08-17 §
22:27 <legoktm> deploying https://gerrit.wikimedia.org/r/305408 [releng]
21:33 <cscott> updated OCG to version e3e0fd015ad8fdbf9da1838c830fe4b075c59a29 [releng]
21:28 <bd808> restarted salt-minion on deployment-pdf02 [releng]
21:26 <bd808> restarted salt-minion on deployment-pdf01 [releng]
21:15 <cscott> starting OCG deploy to beta [releng]
14:10 <gehel> upgrading elasticsearch to 2.3.4 on deployment-logstash2.deployment-prep.eqiad.wmflabs [releng]
13:28 <gehel> upgrading elasticsearch to 2.3.4 on deployment-elastic*.deployment-prep + JVM upgrade [releng]
2016-08-16 §
23:10 <thcipriani> max_servers at 6, seeing 6 allocated instances, still seeing 403 already used 10 of 10 instances :(( [releng]
22:37 <thcipriani> restarting nodepool, bumping max_servers to match up with what openstack seems willing to allocate (6) [releng]
09:06 <Amir1> removing ores-related-cherry-picked commits from deployment-puppetmaster [releng]
2016-08-15 §
21:30 <thcipriani> update scap on beta to 3.2.3-1 bugfix release [releng]
02:30 <bd808> Forced a zuul restart -- https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart [releng]
02:23 <bd808> Lots and lots of "AttributeError: 'NoneType' object has no attribute 'name'" errors in /var/log/zuul/zuul.log [releng]
02:21 <bd808> nodepool delete 301068 [releng]
02:20 <bd808> nodepool delete 301291 [releng]
02:20 <bd808> nodepool delete 301282 [releng]
02:19 <bd808> nodepool delete 301144 [releng]
02:11 <bd808> nodepool delete 299641 [releng]
02:11 <bd808> nodepool delete 278848 [releng]
02:08 <bd808> Aug 15 02:07:48 labnodepool1001 nodepoold[24796]: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) [releng]
2016-08-13 §
23:16 <Amir1> cherry-picking 304678/1 into the puppetmaster [releng]