|
2013-06-28
§
|
| 23:59 |
<jvandavier> |
jvandavier is the best |
[production] |
| 23:58 |
<K4-713> |
updated payments cluster to eee8689ab646 |
[production] |
| 20:54 |
<gwicke> |
updated Parsoid to 454f87a |
[production] |
| 19:19 |
<mutante> |
attempting clean stop and restart of snmptt service on neon |
[production] |
| 19:07 |
<mutante> |
deleting /var/spool/snmptt/ files on neon running out of inodes |
[production] |
| 17:56 |
<^demon> |
restarting gerrit, bringing back missing documentation lost during upgrade |
[production] |
| 17:32 |
<mutante> |
upgrading binutils and libldap on bast1001 |
[production] |
| 17:19 |
<mutante> |
installing some pkg upgrades on fenari |
[production] |
| 16:55 |
<paravoid> |
cause of brief outage: operator error caused restart of varnish on all upload caches eqiad, cascaded into a swift overload, cascaded into imagescaler outage & page |
[production] |
| 16:38 |
<LeslieCarr> |
restarting snmptrapd and smptt on neon |
[production] |
| 16:27 |
<hashar> |
created operations/debs/jenkins-debian-glue and force pushed upstream sources from https://github.com/mika/jenkins-debian-glue |
[production] |
| 15:45 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf9 |
[production] |
| 14:05 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jun 28 14:05:42 UTC 2013 |
[production] |
| 13:57 |
<LocalisationUpdate> |
completed (1.22wmf9) at Fri Jun 28 13:56:55 UTC 2013 |
[production] |
| 13:51 |
<LocalisationUpdate> |
completed (1.22wmf8) at Fri Jun 28 13:51:49 UTC 2013 |
[production] |
| 13:43 |
<reedy> |
Finished syncing Wikimedia installation... : Scaptastic |
[production] |
| 13:41 |
<reedy> |
Started syncing Wikimedia installation... : Scaptastic |
[production] |
| 13:39 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Everything back from wmf7 to wmf8 |
[production] |
| 13:38 |
<reedy> |
synchronized php-1.22wmf9/extensions/Wikibase |
[production] |
| 13:27 |
<reedy> |
Finished syncing Wikimedia installation... : Ensure all l10n caches are built and properly built |
[production] |
| 13:21 |
<reedy> |
Started syncing Wikimedia installation... : Ensure all l10n caches are built and properly built |
[production] |
| 13:20 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 13:15 |
<hashar> |
jenkins: reducing # of executor on master from 4 to 2 |
[production] |
| 12:23 |
<hashar> |
jenkins: migrating mwext-Wikibase-client-tests and mwext-Wikibase-repo-tests from master to slaves |
[production] |
| 09:56 |
<hashar> |
gallium : updating some files under /srv/org to belong to jenkins-slave:jenkins-slave instead of jenkins. |
[production] |
| 09:23 |
<hashar> |
jenkins: made the .erb linting jobs to be voting {{gerrit|70994}} |
[production] |
| 06:27 |
<tstarling> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to wmf8 |
[production] |
| 05:06 |
<tstarling> |
synchronized wmf-config/CommonSettings.php 're-enabling GeoData' |
[production] |
| 05:04 |
<tstarling> |
rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 04:59 |
<Tim> |
reverting to 1.22wmf7 |
[production] |
| 04:57 |
<tstarling> |
synchronized wmf-config/CommonSettings.php 'disable GeoData' |
[production] |
| 04:54 |
<RoanKattouw> |
sync-dir php-1.22wmf8/cache/l10n |
[production] |
| 04:54 |
<Krinkle> |
catrope Removing that extension from extension-list |
[production] |
| 04:53 |
<Krinkle> |
Cause from GeoData "coordinates" magic word. Deployed through broken i18n stuff from WikibaseDataModel deployment earlier that caused fatal during the scap just now |
[production] |
| 04:53 |
<Krinkle> |
About 20 minutes ago the site went down for logged in users (and anon users hitting an uncached page) throwing HTTP 500 |
[production] |
| 04:53 |
<catrope> |
synchronized wmf-config/ExtensionMessages-1.22wmf8.php |
[production] |
| 04:52 |
<tstarling> |
synchronized wmf-config/ExtensionMessages-1.22wmf9.php |
[production] |
| 04:50 |
<catrope> |
Finished syncing Wikimedia installation... : Trying to fix the site |
[production] |
| 04:46 |
<catrope> |
Started syncing Wikimedia installation... : Trying to fix the site |
[production] |
| 04:46 |
<RoanKattouw> |
Running scap in an attempt to fix things |
[production] |
| 04:45 |
<catrope> |
synchronized wmf-config/ExtensionMessages-1.22wmf8.php |
[production] |
| 04:30 |
<catrope> |
Finished syncing Wikimedia installation... : Updating VisualEditor to master |
[production] |
| 04:21 |
<catrope> |
Started syncing Wikimedia installation... : Updating VisualEditor to master |
[production] |
| 02:18 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jun 28 02:18:38 UTC 2013 |
[production] |