3551-3600 of 7884 results (25ms)
2016-02-26 §
11:53 <hashar> memcached process on deployment-memc02 seems to have a nice leak of socket usages (from lost) and plainly refuse connections (bunch of CLOSE_WAIT) T128177 [releng]
11:53 <hashar> memcached process on deployment-memc02 seems to have a nice leak of socket usages (from lost) and plainly refuse connections (bunch of CLOSE_WAIT) [releng]
11:40 <hashar> deployment-memc04 find /etc/apt -name '*proxy' -delete (prevented apt-get update) [releng]
11:26 <hashar> beta: salt -v '*' cmd.run 'apt-get -y install ruby-msgpack' . I am tired of seeing puppet debug messages: "Debug: Failed to load library 'msgpack' for feature 'msgpack'" [releng]
11:24 <hashar> puppet keep restarting nutcracker apparently T128177 [releng]
11:20 <hashar> Memcached error for key "enwiki:flow_workflow%3Av2%3Apk:63dc3cf6a7184c32477496d63c173f9c:4.8" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY [releng]
2016-02-25 §
22:38 <hashar> beta: maybe deployment-jobunner01 is processing jobs a bit faster now. Seems like hhvm went wild [releng]
22:23 <hashar> beta: jobrunner01 had apache/hhvm killed somehow .... Blame me [releng]
21:56 <hashar> beta: stopped jobchron / jobrunner on deployment-jobrunner01 and restarting them by running puppet [releng]
21:49 <hashar> beta did a git-deploy of jobrunner/jobrunner hoping to fix puppet run on deployment-jobrunner01 and apparently it did! T126846 [releng]
11:21 <hashar> deleting workspace /mnt/jenkins-workspace/workspace/browsertests-Wikidata-WikidataTests-linux-firefox-sauce on slave-trusty-1015 [releng]
10:08 <hashar> Jenkins upgraded T128006 [releng]
01:44 <legoktm> deploying https://gerrit.wikimedia.org/r/273170 [releng]
01:39 <legoktm> deploying https://gerrit.wikimedia.org/r/272955 (undeployed) and https://gerrit.wikimedia.org/r/273136 [releng]
01:37 <legoktm> deploying https://gerrit.wikimedia.org/r/273136 [releng]
00:31 <thcipriani> running puppet on beta to update scap to latest packaged version: sudo salt -b '10%' -G 'deployment_target:scap/scap' cmd.run 'puppet agent -t' [releng]
00:20 <thcipriani> deployment-tin not accepting jobs for some time, ran through https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update, is back now [releng]
2016-02-24 §
19:54 <legoktm> legoktm@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki [releng]
18:30 <bd808> "configuration file '/etc/nutcracker/nutcracker.yml' syntax is invalid" [releng]
18:27 <bd808> nutcracker dead on mediawiki01; investigating [releng]
17:20 <hashar> Deleted Nodepool instances so new ones get to use the new snapshot ci-jessie-wikimedia-1456333979 [releng]
17:12 <hashar> Refreshing nodepool snapshot. Been stall since Feb 15th T127755 [releng]
17:01 <bd808> https://wmflabs.org/sal/releng missing SAL data since 2016-02-20T20:19 due to bot crash; needs to be backfilled from wikitech data (T127981) [releng]
2016-02-20 §
20:19 <Krinkle> beta-code-update-eqiad job repeatedly stuck at "IRC notifier plugin" [releng]
19:29 <Krinkle> beta-code-update-eqiad broken because deployment-tin:/srv/mediawiki-staging/php-master/extensions/MobileFrontend/includes/MobileFrontend.hooks.php was modified on the server without commit [releng]
19:22 <Krinkle> Various beta-mediawiki-config-update-eqiad jobs have been stuck 'queued' for > 24 hours [releng]
2016-02-19 §
12:09 <hashar> killed https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ been running for 13 hours. Blocked because slave went offline due to labs reboots yesterday [releng]
10:15 <hashar> Creating a bunch of repository in GitHub to fix Gerrit replication errors [releng]
2016-02-18 §
21:04 <legoktm> deploying https://gerrit.wikimedia.org/r/271600 [releng]
19:20 <legoktm> deploying https://gerrit.wikimedia.org/r/271583 and https://gerrit.wikimedia.org/r/271581, both no-ops [releng]
18:14 <legoktm> deploying https://gerrit.wikimedia.org/r/271012 [releng]
17:36 <legoktm> deploying https://gerrit.wikimedia.org/r/271555 [releng]
16:01 <hashar> deleting instance integration-slave-precise-1003 think we have enough precise slaves [releng]
10:44 <hashar> Nodepool: JenkinsException: Could not parse JSON info for server[https://integration.wikimedia.org/ci/] [releng]
2016-02-17 §
21:11 <jzerebecki> reloading zuul for e11a9ff..d0914a7 [releng]
20:46 <jzerebecki> reloading zuul for 52b90b2..e11a9ff [releng]
18:59 <jzerebecki> updating chery-pick https://gerrit.wikimedia.org/r/#/c/204528/15 on integration-puppetmaster T126699 [releng]
18:11 <jzerebecki> updated cherry-pick https://gerrit.wikimedia.org/r/#/c/204528/14 on integration-puppetmaster T126699 [releng]
15:32 <jzerebecki> reloading zuul for e945e92..52b90b2 [releng]
15:10 <jzerebecki> reloading zuul for bba1873..e945e92 [releng]
14:44 <hashar> On Trusty slaves, reducing number of executors from 4 to 3 to leave room for Mysql/Java/Lua etc [releng]
07:36 <legoktm> deploying https://gerrit.wikimedia.org/r/271201 [releng]
01:01 <yuvipanda> attempting to turn off NFS on 52 instances on deployment-prep project [releng]
2016-02-16 §
23:22 <yuvipanda> new instances on deployment-prep no longer get NFS because of https://wikitech.wikimedia.org/w/index.php?title=Hiera%3ADeployment-prep&type=revision&diff=311783&oldid=311781 [releng]
23:18 <hashar> jenkins@gallium find /var/lib/jenkins/config-history/nodes -maxdepth 1 -type d -name 'ci-jessie*' -exec rm -vfR {} \; [releng]
23:17 <hashar> Jenkins accepting slave creations again. Root cause is /var/lib/jenkins/config-history/nodes/ has reached the 32k inode limit. [releng]
23:14 <hashar> Jenkins: Could not create rootDir /var/lib/jenkins/config-history/nodes/ci-jessie-wikimedia-34969/2016-02-16_22-40-23 [releng]
23:02 <hashar> Nodepool can not authenticate with Jenkins anymore. Thus it can not add slaves it spawned. [releng]
22:56 <hashar> contint: Nodepool instances pool exhausted [releng]
21:14 <andrewbogott> deployment-logstash2 migration finished [releng]