|
2015-04-03
ยง
|
| 19:42 |
<twentyafterfour> |
Synchronized php-1.25wmf24/extensions/OpenStackManager/nova/OpenStackNovaUser.php: sync security patch (duration: 00m 12s) |
[production] |
| 18:55 |
<mutante> |
restarted grrrit-wm for config change |
[production] |
| 18:40 |
<ori> |
Restarted nutcracker on HHVM and mw1147 and repooled |
[production] |
| 18:35 |
<ori> |
Depooled mw1147. Spamming fluorine:/a/mw-log/memcache-serious.log. Some nutcracker issue most likely. |
[production] |
| 17:49 |
<ori> |
Synchronized wmf-config/CommonSettings.php: I9c4de264: Send server side eventlogging logs to eventlog1001 instead of vanadium (duration: 00m 11s) |
[production] |
| 15:14 |
<kartik> |
Synchronized php-1.25wmf24/extensions/ContentTranslation: (no message) (duration: 00m 14s) |
[production] |
| 15:14 |
<kartik> |
Synchronized php-1.25wmf23/extensions/ContentTranslation: (no message) (duration: 00m 17s) |
[production] |
| 14:49 |
<hashar> |
integration-slave-jessie-1001 : manually installed jenkins-debian-glue Debian packages. It is pending upload by ops to apt.wikimedia.org {{bug|T95006}} |
[releng] |
| 12:56 |
<hashar> |
installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-101* instances |
[releng] |
| 12:56 |
<hashar> |
installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-1011.eqiad.wmflabs |
[releng] |
| 12:35 |
<hashar> |
Switching Jessie slave from role::ci::slave::labs::common to role::ci::slave::labs which will bring a whole lot of packages and break |
[releng] |
| 12:28 |
<hashar> |
integration-slave-jessie-1001 applying role::ci::slave::labs::common to pool it as a very basic Jenkins slave |
[releng] |
| 12:19 |
<hashar> |
enabled puppetmaster::autosigner on integration-puppetmaster |
[releng] |
| 11:58 |
<hashar> |
Applied role::ci::slave::labs on integration-slave-precise-101[1-4] that Timo created earlier |
[releng] |
| 11:58 |
<hashar> |
Cherry picked a couple patches to fix puppet Package[] definitions issues |
[releng] |
| 11:49 |
<hashar> |
made integration-puppetmaster to self update its puppet clone |
[releng] |
| 11:42 |
<hashar> |
recreating integration-slave-precise-1011 stalled with a puppet oddity related to Package['gdb'] defined twice {{bug|T94917}} |
[releng] |
| 11:30 |
<hashar> |
integration-puppetmaster migrated down to Precise |
[releng] |
| 11:23 |
<hashar> |
rebooting integration-publisher : cant ssh to it |
[releng] |
| 11:14 |
<springle> |
Synchronized wmf-config/db-eqiad.php: db1049 to normal load (duration: 00m 11s) |
[production] |
| 10:37 |
<hashar> |
disabled some hiera configuration related to puppetmaster. |
[releng] |
| 10:37 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1049, warm up (duration: 00m 12s) |
[production] |
| 10:32 |
<paravoid> |
depooled mw1234 |
[production] |
| 10:22 |
<hashar> |
Created instance i-00000a4a with image "ubuntu-12.04-precise" and hostname i-00000a4a.eqiad.wmflabs. |
[releng] |
| 10:21 |
<hashar> |
downgrading integration-puppetmaster from Trusty to Precise https://phabricator.wikimedia.org/T94927 |
[releng] |
| 10:20 |
<paravoid> |
staggered restart of the API cluster (sans mw1234, left for further debugging) |
[production] |
| 09:32 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 20s) |
[production] |
| 09:24 |
<springle> |
mw1114 critical, no ssh, no console, powercycle |
[production] |
| 09:19 |
<springle> |
tin sync-file: mw1114.eqiad.wmnet returned [-15] |
[production] |
| 09:18 |
<springle> |
Synchronized wmf-config/db-eqiad.php: reduce db1049 load (duration: 06m 26s) |
[production] |
| 05:42 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/200744 |
[releng] |
| 04:56 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Apr 3 04:55:08 UTC 2015 (duration 55m 7s) |
[production] |
| 03:58 |
<Krinkle> |
Jobs were throwing NOT_RECOGNISED. Relaunched Gearman. Jobs are now happy again. |
[releng] |
| 03:51 |
<Krinkle> |
Jenkins is unable to re-establish Gearman connection. Have to force restart Jenkins master. |
[releng] |
| 03:44 |
<greg-g> |
*unable |
[releng] |
| 03:44 |
<Krinkle> |
References to past hour of builds have been restored. But Jenkins is still enable to make new references properly. New builds are 404'ing the same way. |
[releng] |
| 03:42 |
<Krinkle> |
Reloading Jenking config repaired the broken references. Build urls are now resolving again. |
[releng] |
| 03:31 |
<mattflaschen> |
Synchronized php-1.25wmf24/includes/libs/normal/UtfNormalUtil.php: Fix UtfNormal shim so account creations work (duration: 00m 12s) |
[production] |
| 03:29 |
<mattflaschen> |
Synchronized php-1.25wmf24/includes/libs/normal/UtfNormalUtil.php: Fix UtfNormal shim so account creations work (duration: 00m 12s) |
[production] |
| 03:26 |
<Krinkle> |
Reloading Jenkins configuration from disk to mitigate |
[releng] |
| 03:18 |
<Krinkle> |
The failure started at 03:03 exactly. The newer build metadata exists at /var/lib/jenkins/jobs/:jobname/builds/:nr, but the jobs/*/last*Build symlinks are no longer updated. |
[releng] |
| 03:03 |
<LocalisationUpdate> |
completed (1.25wmf24) at 2015-04-03 03:02:30+00:00 |
[production] |
| 02:59 |
<l10nupdate> |
Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 00s) |
[production] |
| 02:47 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/201644 |
[releng] |
| 02:38 |
<LocalisationUpdate> |
completed (1.25wmf23) at 2015-04-03 02:37:21+00:00 |
[production] |
| 02:31 |
<l10nupdate> |
Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 09m 08s) |
[production] |
| 02:03 |
<YuviPanda> |
restarted hhvm on mw1209 |
[production] |
| 02:02 |
<YuviPanda> |
restarted hhvm on mw1249 and mw1065 |
[production] |
| 01:54 |
<catrope> |
Synchronized w: (no message) (duration: 00m 12s) |
[production] |
| 01:30 |
<ori> |
Synchronized php-1.25wmf24/extensions/ConfirmEdit: 7cb7ef4e6f: Update ConfirmEdit for Id4798364d (duration: 00m 12s) |
[production] |