|
2013-05-10
§
|
| 09:17 |
<hashar> |
gallium: dumping java heap of Jenkins using jmap -dump:format=b,file=/root/jenkins-bug48025.jmap -F 24508 |
[production] |
| 08:33 |
<hashar> |
Jenkins hit by the evil bug again {{bug|48025}}. Taking as much trace as possible before restarting it |
[production] |
| 04:46 |
<paravoid> |
swift: pushing new rings, set weight 0 for ms-be5 sdb1, ms-be11 sdh1 (broken disks); balance 999.99, needs further rebalance |
[production] |
| 03:37 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri May 10 03:37:41 UTC 2013 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.22wmf3) at Fri May 10 02:15:49 UTC 2013 |
[production] |
| 00:14 |
<pgehres> |
synchronized wmf-config 'Actually enabling wgCentralAuthAutoMigrate' |
[production] |
|
2013-05-09
§
|
| 23:34 |
<mwalker> |
synchronized wmf-config/InitialiseSettings.php 'Enabling siteNotice div on all alpha and beta mobile sites' |
[production] |
| 23:26 |
<pgehres> |
synchronized php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php 'Moving one more CentralAuth log line' |
[production] |
| 23:21 |
<mwalker> |
synchronized php-1.22wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master for some bugfixes' |
[production] |
| 23:06 |
<pgehres> |
synchronized php-1.22wmf3/extensions/CentralAuth/ 'Updates to CentralAuth' |
[production] |
| 21:54 |
<hashar> |
Jenkins / Zuul is back up (for now). |
[production] |
| 21:44 |
<hashar> |
restarted Jenkins to enable access logs in /var/log/jenkins/access.log {{gerrit|63068}} |
[production] |
| 21:42 |
<hashar> |
forced Zuul to stop. All events have been dropped :( |
[production] |
| 21:31 |
<hashar> |
restarting Zuul to make it query Jenkins directly without SSL + apache proxy {{gerrit|63069}}. |
[production] |
| 21:29 |
<Jeff_Green> |
pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed |
[production] |
| 21:28 |
<^demon> |
gerrit repositories will now recursively merge |
[production] |
| 21:16 |
<olivneh> |
synchronized php-1.22wmf3/extensions/WikimediaMessages 'Updating WikimediaMessages to master' |
[production] |
| 20:47 |
<olivneh> |
synchronized php-1.22wmf3/extensions/GuidedTour 'Actually updating GuidedTour to master' |
[production] |
| 20:46 |
<olivneh> |
synchronized php-1.22wmf3/extensions/EventLogging 'Updating GuidedTour to master' |
[production] |
| 20:45 |
<olivneh> |
synchronized php-1.22wmf3/extensions/EventLogging 'Updating EventLogging to master' |
[production] |
| 20:45 |
<olivneh> |
synchronized php-1.22wmf3/extensions/ConfirmEdit 'Updating ConfirmEdit to master' |
[production] |
| 20:20 |
<hashar> |
restarting Jenkins. |
[production] |
| 20:20 |
<hashar> |
Jenkins died :/ |
[production] |
| 19:16 |
<mutante> |
fixing file permissions in /a/e3/E3Analysis repo on stat1001 |
[production] |
| 19:09 |
<mutante> |
restarted Apache and then puppetmaster on stafford, that revived it |
[production] |
| 18:55 |
<mutante> |
attempting to restart broken puppetmaster on stafford |
[production] |
| 18:27 |
<pgehres> |
synchronized wmf-config/InitialiseSettings.php 'Enabling CentralAuth debug log' |
[production] |
| 18:00 |
<pgehres> |
synchronized wmf-config/InitialiseSettings.php 'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth' |
[production] |
| 15:56 |
<cmjohnson1> |
powering down barium /relocate server to c1 |
[production] |
| 15:26 |
<anomie> |
synchronized wmf-config/CommonSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' |
[production] |
| 15:26 |
<anomie> |
synchronized wmf-config/InitialiseSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' |
[production] |
| 15:26 |
<anomie> |
synchronized wmf-config/InitialiseSettings-labs.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' |
[production] |
| 11:24 |
<hashar> |
upgrading Zuul to latest master (makes report times friendlier) |
[production] |
| 08:54 |
<hashar> |
Jenkins apparently working fine. I have upgraded a few plugins, that seems to have fixed the Gerrit jobs. |
[production] |
| 08:22 |
<hashar> |
restarting Jenkins again |
[production] |
| 08:20 |
<hashar> |
upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris) |
[production] |
| 07:39 |
<hashar> |
restarting jenkins. All its http thread are locked :/ |
[production] |
| 07:25 |
<hashar> |
Jenkins HTTP interface died :/ |
[production] |
| 03:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu May 9 03:39:55 UTC 2013 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.22wmf3) at Thu May 9 02:15:54 UTC 2013 |
[production] |
|
2013-05-08
§
|
| 22:26 |
<RobH> |
authdns update to migrate racktables from hooper to magnesium. both servers are already using db1001 as master, which is why hooper is slow to load racktables data |
[production] |
| 22:23 |
<mholmquist> |
synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php 'Fixing Special:Preferences for wikis with UploadWizard' |
[production] |
| 21:44 |
<mholmquist> |
Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates |
[production] |
| 21:36 |
<mholmquist> |
Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates |
[production] |
| 21:06 |
<Krinkle> |
Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276) |
[production] |
| 20:20 |
<mwalker> |
synchronized php-1.22wmf3/extensions/CentralNotice/ 'CentralNotice security bugfix 48255' |
[production] |
| 20:15 |
<Jeff_Green> |
authdns-update to swap in grosley for dead aluminium |
[production] |
| 19:41 |
<RobH> |
magnesium is future racktables host, disregard errors for now |
[production] |
| 19:40 |
<cmjohnson1> |
accidently removed power from magnesium |
[production] |
| 19:00 |
<Reedy> |
Ensuring securepoll database tables exist on all wikis |
[production] |