|
2015-03-03
§
|
| 18:48 |
<Krinkle> |
New builds run fine, but there's 30 stuck builds occupying executors. Their output is finished and Zuul/Gerrit got the event already, but they won't die. |
[releng] |
| 18:30 |
<Krinkle> |
The stuck builds have a line of "Finished: .." in them, but are still showing a loader spinner on the bottom of their build log 10 minutes later. |
[releng] |
| 18:28 |
<Krinkle> |
Lots of Jenkins builds are stuck half-way executing. No clear cause. Everything seems up. |
[releng] |
| 17:18 |
<Krinkle> |
Reloading Zuul to deploy Icad0a26dc8 and Icac172b16 |
[releng] |
| 15:39 |
<hashar> |
cancelled logrotate update of all jobs since that seems to kill the Jenkins/Zuul gearman connection. Probably because all jobs are registered on each config change. |
[releng] |
| 15:31 |
<hashar> |
updating all jobs in Jenkins based on PS2 of https://gerrit.wikimedia.org/r/194109 |
[releng] |
| 10:56 |
<hashar> |
Created instance i-000008fb with image "ubuntu-14.04-trusty" and hostname i-000008fb.eqiad.wmflabs. |
[releng] |
| 10:52 |
<hashar> |
deleting integration-puppetmaster to recreate it with a new image {bug|T87484} . Will have to reapply I5335ea7cbfba33e84b3ddc6e3dd83a7232b8acfd and I30e5bfeac398e0f88e538c75554439fe82fcc1cf |
[releng] |
| 03:47 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts 05a5593..1e64ed9 |
[releng] |
| 01:11 |
<marxarelli> |
gzip'd /var/log/account/pacct.0 on deployment-bastion to free space |
[releng] |
|
2015-02-26
§
|
| 23:24 |
<Krinkle> |
integration-puppetmaster /var disk is full (1.8 of 1.9GB) - /var/log/puppet/reports is 1.1GB - purging |
[releng] |
| 23:23 |
<Krinkle> |
Puppet failing on new instances due to "Error 400 on SERVER: cannot generate tempfile `/var/lib/puppet/yaml/" |
[releng] |
| 13:27 |
<Krinkle> |
Provisioning the new integration-slave12xx and integration-slave14xx instances |
[releng] |
| 11:53 |
<YuviPanda> |
created deployment-parsoid01-test to test patch to use role::parsoid on labs |
[releng] |
| 05:05 |
<legoktm> |
deployed https://gerrit.wikimedia.org/r/192980 |
[releng] |
| 03:48 |
<Krinkle> |
Creating integration-slave1201,02,03,04 and integration-slave1401,02,03,04,05 per T74011 (not yet setup/provisioned, keep depooled) |
[releng] |
| 03:39 |
<Krinkle> |
Cleaned up and re-pooled integration-slave1006 (was depooled since yesterday) |
[releng] |
| 03:39 |
<Krinkle> |
Cleaned up and re-pooled integration-slave1007 and integration-slave1008 (was auto-depooled by Jenkins) |
[releng] |
| 01:54 |
<Krinkle> |
integration-slave1007 and integration-slave1008 were auto-deplooed due to main disk (/ and its /tmp) being < 900 MB free |
[releng] |
| 01:20 |
<legoktm> |
actually deployed https://gerrit.wikimedia.org/r/192772 this time |
[releng] |
| 01:16 |
<legoktm> |
deployed https://gerrit.wikimedia.org/r/192772 |
[releng] |