651-700 of 3402 results (17ms)
2015-04-16 §
22:07 <Krinkle> Rebooted integration-slave-trusty-1015 (experimenting with libeatmydata) [releng]
18:31 <Krinkle> Rebooting integration-slave-precise-1012 and integration-slave-trusty-1012 [releng]
17:57 <Krinkle> Repooled instances. Converstion of mysql.datadir to tmpfs worked, but puppet run has errors. Coren and Krinkle working on it. https://gerrit.wikimedia.org/r/#/c/204528/ (T96230) [releng]
17:22 <Krinkle> Gracefully depool integration slaves to deploy https://gerrit.wikimedia.org/r/#/c/204528/ (T96230) [releng]
14:35 <thcipriani> running dpkg --configure -a on deployment-bastion to correct puppet failures [releng]
2015-04-15 §
23:21 <Krinkle> beta-update-databases-eqiad stuck waiting for executors on a node that has plenty executors available [releng]
21:15 <hashar> Jenkins browser test jobs sometime deadlock because of the IRC notification plugin https://phabricator.wikimedia.org/T96183 [releng]
20:34 <hashar> hard restarting Jenkins [releng]
19:24 <Krinkle> Aborting browser tests jobs. Stuck for over 5 hours. [releng]
19:24 <Krinkle> Aborting beta-scap-eqiad. Has been stuck for 2 hours on "Notifying IRC" after "Connection time out" from scap. [releng]
08:22 <hashar> restarted Jenkins [releng]
08:20 <hashar> Exception in thread "RequestHandlerThread[#2]" java.lang.OutOfMemoryError: Java heap space [releng]
08:16 <hashar> Jenkins process went wild taking all CPU busy on gallium [releng]
2015-04-14 §
20:43 <legoktm> starting SULF on beta cluster [releng]
20:42 <marktraceur> stopping all beta jobs, aborting running (and stuck) beta DB update, kicking bastion, to try and get beta to update [releng]
19:49 <Krinkle> All systems go. [releng]
19:48 <Krinkle> Jenkins configuration panel won't load ("Loading..." stays indefine, "Uncaught TypeError: Cannot convert to object at prototype.js:195") [releng]
19:46 <Krinkle> Jenkins restarted. Relaunching Gearman [releng]
19:42 <Krinkle> Jenkins still unable to obtain Gearman connection. (HTTP 503 error from /configure). Have to force restart Jenkins. [releng]
19:42 <Krinkle> deployment-bastion jobs were stuck. marktraceur cancelled queue and relaunched slave. Now processing again. [releng]
15:27 <Krinkle> puppetmaster: Re-apply I05c49e5248cb operations/puppet patch to re-fix T91524. Somehow the patch got lost. [releng]
08:46 <hashar> does qa-morebots works ? [releng]
2015-04-13 §
20:14 <Krinkle> Restarting Zuul, Jenkins and aborting all builds. Everything got stuck following NFS outage in lab [releng]
17:01 <legoktm> deploying https://gerrit.wikimedia.org/r/203858 [releng]
13:56 <Krinkle> Delete old integration-slave1001...1004 (T94916) [releng]
10:43 <hashar> reducing number of executors on Precise instances from 5 to 4 and on Trusty instances from 6 to 4. The Jenkins scheduler tends to assign the unified jobs to the same slave which overload a single slave while others are idling. [releng]
10:43 <hashar> reducing number of executors from 5 to 4 [releng]
08:46 <hashar> jenkins removed #wikimedia-qa IRC channel from the global configuration [releng]
08:42 <hashar> kill -9 jenkins causes it was stuck in some deadlock related to the IRC plugin :( [releng]
08:34 <zeljkof> restarting stuck Jenkins [releng]
2015-04-12 §
23:58 <bd808> sudo ln -s /srv/l10nupdate/mediawiki /var/lib/l10nupdate/mediawiki on deployment-bastion [releng]
23:11 <greg-g> 0bytes left on /var on deployment-bastion [releng]
2015-04-11 §
23:13 <legoktm> deploying https://gerrit.wikimedia.org/r/203628 [releng]
22:58 <legoktm> deploying https://gerrit.wikimedia.org/r/203619 & https://gerrit.wikimedia.org/r/203626 [releng]
06:13 <legoktm> deployed https://gerrit.wikimedia.org/r/203520 [releng]
05:49 <legoktm> deploying https://gerrit.wikimedia.org/r/203519 https://gerrit.wikimedia.org/r/203516 https://gerrit.wikimedia.org/r/203518 [releng]
2015-04-10 §
13:50 <Krinkle> Pool integration-slave-precise-1012..integration-slave-precise-1014 [releng]
11:43 <hashar> Filled https://phabricator.wikimedia.org/T95675 to migrate "Global-Dev Dashboard Data" to JJB/Zuul [releng]
11:40 <Krinkle> Deleting various Jenkins jobs that can be safely deleted (recently removed from jjb-config). Will report the rest to T91410 for inspection. [releng]
11:29 <Krinkle> Fixed job "Global-Dev Dashboard Data" to be restricted to node "gallium" because it fails to connect to gp.wmflabs.org from lanthanum 1/2 builds. [releng]
11:26 <Krinkle> Re-established Gearman connection from Jenkins [releng]
11:20 <Krinkle> Jenkins unable to re-establish Gearman connection. Full restart. [releng]
10:39 <Krinkle> Deleting the old integration1401...integration1405 instances. They've been depooled for 24h and their replacements are OK. This is to free up quota to create new Precise instances. [releng]
10:35 <Krinkle> Creating integration-slave-precise-1012...integration-slave-precise-1014 [releng]
10:31 <Krinkle> Pool integration-slave-precise-1011 [releng]
09:02 <hashar> integration: Refreshed Zuul packages under /home/hashar [releng]
08:57 <Krinkle> Fixed puppet failure for missing Zuul package on integration-dev by applying patch-integration-slave-trusty.sh [releng]
2015-04-09 §
20:11 <mutante> fixed apt sources lists on deployment-bastion (T95541) [releng]
19:50 <legoktm> deployed https://gerrit.wikimedia.org/r/202932 [releng]
17:20 <Krinkle> Creating integration-slave-precise-1011 [releng]