|
2015-04-03
§
|
| 21:31 |
<Krinkle> |
Upgrading npm from v2.4.1 to v2.7.5 (rolling, slave by slave graceful) |
[releng] |
| 21:11 |
<^d> |
puppet re-enabled on staging-palladium, running fine again |
[releng] |
| 21:05 |
<Krinkle> |
Delete unfinished/unpoooled instances integration-slave-precise-1011-1014. (T94916) |
[releng] |
| 14:49 |
<hashar> |
integration-slave-jessie-1001 : manually installed jenkins-debian-glue Debian packages. It is pending upload by ops to apt.wikimedia.org {{bug|T95006}} |
[releng] |
| 12:56 |
<hashar> |
installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-101* instances |
[releng] |
| 12:56 |
<hashar> |
installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-1011.eqiad.wmflabs |
[releng] |
| 12:35 |
<hashar> |
Switching Jessie slave from role::ci::slave::labs::common to role::ci::slave::labs which will bring a whole lot of packages and break |
[releng] |
| 12:28 |
<hashar> |
integration-slave-jessie-1001 applying role::ci::slave::labs::common to pool it as a very basic Jenkins slave |
[releng] |
| 12:19 |
<hashar> |
enabled puppetmaster::autosigner on integration-puppetmaster |
[releng] |
| 11:58 |
<hashar> |
Applied role::ci::slave::labs on integration-slave-precise-101[1-4] that Timo created earlier |
[releng] |
| 11:58 |
<hashar> |
Cherry picked a couple patches to fix puppet Package[] definitions issues |
[releng] |
| 11:49 |
<hashar> |
made integration-puppetmaster to self update its puppet clone |
[releng] |
| 11:42 |
<hashar> |
recreating integration-slave-precise-1011 stalled with a puppet oddity related to Package['gdb'] defined twice {{bug|T94917}} |
[releng] |
| 11:30 |
<hashar> |
integration-puppetmaster migrated down to Precise |
[releng] |
| 11:23 |
<hashar> |
rebooting integration-publisher : cant ssh to it |
[releng] |
| 10:37 |
<hashar> |
disabled some hiera configuration related to puppetmaster. |
[releng] |
| 10:22 |
<hashar> |
Created instance i-00000a4a with image "ubuntu-12.04-precise" and hostname i-00000a4a.eqiad.wmflabs. |
[releng] |
| 10:21 |
<hashar> |
downgrading integration-puppetmaster from Trusty to Precise https://phabricator.wikimedia.org/T94927 |
[releng] |
| 05:42 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/200744 |
[releng] |
| 03:58 |
<Krinkle> |
Jobs were throwing NOT_RECOGNISED. Relaunched Gearman. Jobs are now happy again. |
[releng] |
| 03:51 |
<Krinkle> |
Jenkins is unable to re-establish Gearman connection. Have to force restart Jenkins master. |
[releng] |
| 03:44 |
<greg-g> |
*unable |
[releng] |
| 03:44 |
<Krinkle> |
References to past hour of builds have been restored. But Jenkins is still enable to make new references properly. New builds are 404'ing the same way. |
[releng] |
| 03:42 |
<Krinkle> |
Reloading Jenking config repaired the broken references. Build urls are now resolving again. |
[releng] |
| 03:26 |
<Krinkle> |
Reloading Jenkins configuration from disk to mitigate |
[releng] |
| 03:18 |
<Krinkle> |
The failure started at 03:03 exactly. The newer build metadata exists at /var/lib/jenkins/jobs/:jobname/builds/:nr, but the jobs/*/last*Build symlinks are no longer updated. |
[releng] |
| 02:47 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/201644 |
[releng] |
| 00:31 |
<greg-g> |
rm 'd .gitignore in /srv/mediawiki-staging/php-master/skins due to https://gerrit.wikimedia.org/r/#/c/200307/ clashing with a local untracked version |
[releng] |
|
2015-04-02
§
|
| 22:56 |
<Krinkle> |
New integration-slave-precise-101x are unfinished and must remain depooled. See T94916. |
[releng] |
| 22:53 |
<Krinkle> |
Most puppet failures blocking T94916 may be caused by the fact that intergration-puppetmaster was inadvertently changed to Trusty; puppetmaster version of Trusty is not yet supported by ops |
[releng] |
| 21:41 |
<Krinkle> |
It seems integration-slave-jessie-1001 has role::ci::slave::labs::common instead of role::ci::slave::labs. Intentional? |
[releng] |
| 21:25 |
<Krinkle> |
Re-creating integration-dev-slave-precise in preparation of re-creating precise slaves |
[releng] |
| 14:51 |
<hashar> |
applying role::ci::slave::labs::common on integration-slave-jessie-1001 |
[releng] |
| 14:49 |
<hashar> |
integration: nice thing, newly created instances are automatically made to point to integration-pummetmaster via hiera! Just have to sign the certificate on the master using: puppet ca list ; puppet ca sign i-000xxxx.eqiad.wmflabs |
[releng] |
| 14:42 |
<hashar> |
Created [[Nova_Resource:I-00000a3b.eqiad.wmflabs|integration-slave-jessie-1001]] to try out CI slave on Jessie ([[T94836]]) |
[releng] |
| 14:11 |
<hashar> |
reduced integration-slave1004 executors from 6 to 5 to make it on par with the other precise slaves |
[releng] |
| 14:10 |
<hashar> |
integration-slave100[1-4] are now using Zuul provided by a Debian package as of https://gerrit.wikimedia.org/r/#/c/195272/ PS 16 |
[releng] |
| 14:04 |
<hashar> |
uninstall the pip installed zuul version from Precise labs slaves by doing: pip uninstall zuul && rm /usr/local/bin/zuul* . Switching them all to a Debian package |
[releng] |
| 13:45 |
<hashar> |
pooling back integration-slave1001 and 1002 which are using zuul-cloner provided by a debian package |
[releng] |
| 13:35 |
<hashar> |
reloading Jenkins configuration files from disk to make it knows about a change manually applied to most jobs config.xml files for https://gerrit.wikimedia.org/r/#/c/201451/ |
[releng] |
| 13:01 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/201458 |
[releng] |
| 12:19 |
<hashar> |
preventing job to run on integration-slave1001 by replacing its label with 'DoNotLabelThisSlaveHashar'. Going to install Zuul debian package on it |
[releng] |
| 09:37 |
<hashar> |
rebooting integration-zuul-server homedir seems to be stalled/missing |
[releng] |
| 08:12 |
<hashar> |
upgrading packages on integration-dev |
[releng] |
| 05:14 |
<greg-g> |
and right when I log'd that, things seem to be recovering |
[releng] |
| 05:12 |
<greg-g> |
the shinken alerts about beta cluster issues are due to wmflabs having issues. |
[releng] |