|
2015-04-03
§
|
| 10:21 |
<hashar> |
downgrading integration-puppetmaster from Trusty to Precise https://phabricator.wikimedia.org/T94927 |
[releng] |
| 10:20 |
<paravoid> |
staggered restart of the API cluster (sans mw1234, left for further debugging) |
[production] |
| 09:32 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 20s) |
[production] |
| 09:24 |
<springle> |
mw1114 critical, no ssh, no console, powercycle |
[production] |
| 09:19 |
<springle> |
tin sync-file: mw1114.eqiad.wmnet returned [-15] |
[production] |
| 09:18 |
<springle> |
Synchronized wmf-config/db-eqiad.php: reduce db1049 load (duration: 06m 26s) |
[production] |
| 05:42 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/200744 |
[releng] |
| 04:56 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Apr 3 04:55:08 UTC 2015 (duration 55m 7s) |
[production] |
| 03:58 |
<Krinkle> |
Jobs were throwing NOT_RECOGNISED. Relaunched Gearman. Jobs are now happy again. |
[releng] |
| 03:51 |
<Krinkle> |
Jenkins is unable to re-establish Gearman connection. Have to force restart Jenkins master. |
[releng] |
| 03:44 |
<greg-g> |
*unable |
[releng] |
| 03:44 |
<Krinkle> |
References to past hour of builds have been restored. But Jenkins is still enable to make new references properly. New builds are 404'ing the same way. |
[releng] |
| 03:42 |
<Krinkle> |
Reloading Jenking config repaired the broken references. Build urls are now resolving again. |
[releng] |
| 03:31 |
<mattflaschen> |
Synchronized php-1.25wmf24/includes/libs/normal/UtfNormalUtil.php: Fix UtfNormal shim so account creations work (duration: 00m 12s) |
[production] |
| 03:29 |
<mattflaschen> |
Synchronized php-1.25wmf24/includes/libs/normal/UtfNormalUtil.php: Fix UtfNormal shim so account creations work (duration: 00m 12s) |
[production] |
| 03:26 |
<Krinkle> |
Reloading Jenkins configuration from disk to mitigate |
[releng] |
| 03:18 |
<Krinkle> |
The failure started at 03:03 exactly. The newer build metadata exists at /var/lib/jenkins/jobs/:jobname/builds/:nr, but the jobs/*/last*Build symlinks are no longer updated. |
[releng] |
| 03:03 |
<LocalisationUpdate> |
completed (1.25wmf24) at 2015-04-03 03:02:30+00:00 |
[production] |
| 02:59 |
<l10nupdate> |
Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 06m 00s) |
[production] |
| 02:47 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/201644 |
[releng] |
| 02:38 |
<LocalisationUpdate> |
completed (1.25wmf23) at 2015-04-03 02:37:21+00:00 |
[production] |
| 02:31 |
<l10nupdate> |
Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 09m 08s) |
[production] |
| 02:03 |
<YuviPanda> |
restarted hhvm on mw1209 |
[production] |
| 02:02 |
<YuviPanda> |
restarted hhvm on mw1249 and mw1065 |
[production] |
| 01:54 |
<catrope> |
Synchronized w: (no message) (duration: 00m 12s) |
[production] |
| 01:30 |
<ori> |
Synchronized php-1.25wmf24/extensions/ConfirmEdit: 7cb7ef4e6f: Update ConfirmEdit for Id4798364d (duration: 00m 12s) |
[production] |
| 01:11 |
<catrope> |
Synchronized php-1.25wmf24/extensions/ContentTranslation/modules/campaigns/ext.cx.campaigns.contributionsmenu.js: touch (duration: 00m 13s) |
[production] |
| 01:11 |
<catrope> |
Synchronized php-1.25wmf23/extensions/ContentTranslation/modules/campaigns/ext.cx.campaigns.contributionsmenu.js: touch (duration: 00m 12s) |
[production] |
| 01:08 |
<catrope> |
Synchronized php-1.25wmf23/includes: SWAT (duration: 00m 15s) |
[production] |
| 01:06 |
<catrope> |
Synchronized php-1.25wmf23/autoload.php: SWAT (duration: 00m 12s) |
[production] |
| 01:04 |
<catrope> |
Synchronized php-1.25wmf24/includes: SWAT (duration: 00m 15s) |
[production] |
| 01:03 |
<catrope> |
Synchronized php-1.25wmf24/autoload.php: (no message) (duration: 00m 11s) |
[production] |
| 01:01 |
<catrope> |
Synchronized php-1.25wmf24/extensions/Gather: SWAT (duration: 00m 13s) |
[production] |
| 01:00 |
<ori> |
restart HHVM on mw1120 |
[production] |
| 00:48 |
<catrope> |
Synchronized php-1.25wmf24/extensions/VisualEditor: SWAT (duration: 00m 12s) |
[production] |
| 00:47 |
<catrope> |
Synchronized php-1.25wmf24/extensions/Flow: SWAT (duration: 00m 14s) |
[production] |
| 00:47 |
<catrope> |
Synchronized php-1.25wmf24/extensions/ConfirmEdit: SWAT (duration: 00m 13s) |
[production] |
| 00:47 |
<catrope> |
Synchronized php-1.25wmf23/extensions/Flow: SWAT (duration: 00m 12s) |
[production] |
| 00:47 |
<catrope> |
Synchronized php-1.25wmf23/extensions/ConfirmEdit: SWAT (duration: 00m 11s) |
[production] |
| 00:44 |
<catrope> |
Synchronized php-1.25wmf23/extensions/Gather: SWAT (duration: 00m 11s) |
[production] |
| 00:31 |
<greg-g> |
rm 'd .gitignore in /srv/mediawiki-staging/php-master/skins due to https://gerrit.wikimedia.org/r/#/c/200307/ clashing with a local untracked version |
[releng] |
|
2015-04-02
§
|
| 23:11 |
<mutante> |
temp. disabling puppet on restbase servers |
[production] |
| 22:56 |
<Krinkle> |
New integration-slave-precise-101x are unfinished and must remain depooled. See T94916. |
[releng] |
| 22:53 |
<Krinkle> |
Most puppet failures blocking T94916 may be caused by the fact that intergration-puppetmaster was inadvertently changed to Trusty; puppetmaster version of Trusty is not yet supported by ops |
[releng] |
| 22:50 |
<bd808> |
lots of SYSTEM ERROR responses from nutcracker on mw1147 |
[production] |
| 22:13 |
<greg-g> |
Account creation is broken/not working for either iOS or Android WP apps, investigation in -mobile |
[production] |
| 21:41 |
<Krinkle> |
It seems integration-slave-jessie-1001 has role::ci::slave::labs::common instead of role::ci::slave::labs. Intentional? |
[releng] |
| 21:25 |
<Krinkle> |
Re-creating integration-dev-slave-precise in preparation of re-creating precise slaves |
[releng] |
| 19:37 |
<ori> |
Synchronized wmf-config/InitialiseSettings.php: I3bbf2418d: Set $wgLogoHD for enwiki (duration: 00m 12s) |
[production] |
| 18:52 |
<mutante> |
running puppet on mw2095 - proxy error |
[production] |