|
2013-05-10
§
|
| 18:49 |
<mutante> |
killing and restarting nagios-nrpe-server on a bunch of cp10xx boxes to fix monitoring of varnishncsa |
[production] |
| 18:22 |
<mutante> |
powercycling srv291 |
[production] |
| 18:16 |
<RobH> |
authdns-update to remove blog.wikimediafoundation as it wasnt setup and pointed to decom server |
[production] |
| 17:41 |
<RobH> |
authdnsupdate - removing racktables2 entry |
[production] |
| 17:18 |
<hashar> |
jenkins: cleared all occurrences of an old plugin in build histories. {{bug|48025}}. Jenkins should be fine now. Will monitor tonight. |
[production] |
| 13:20 |
<mark> |
Migrated toolserver routing to cr1-esams and cr2-esams via csw2-esams access |
[production] |
| 12:27 |
<mark> |
Removed knsq23-knsq30 from the text squid config |
[production] |
| 12:24 |
<hashar> |
jenkins: cleaning up all occurrences of an old plugin in build histories. The script run in a tmux on gallium (see https://bugzilla.wikimedia.org/show_bug.cgi?id=48025#c19 ) |
[production] |
| 12:08 |
<mark> |
Depooled knsq23-knsq30 text squid frontends |
[production] |
| 11:52 |
<mark> |
Moved amslvs1 and 2 from csw1-esams to csw2-esams |
[production] |
| 11:52 |
<mark> |
Started PyBal on amslvs1 |
[production] |
| 11:50 |
<mark> |
Stopped PyBal on amslvs1 |
[production] |
| 11:49 |
<mark> |
Reenabled PyBal on amslvs2 |
[production] |
| 11:44 |
<hashar> |
restarting jenkins |
[production] |
| 11:44 |
<mark> |
Disabled Puppet and stopped PyBal on amslvs2 |
[production] |
| 11:27 |
<mark> |
Moved ssl3001-3004 from csw1-esams to csw2-esams |
[production] |
| 10:09 |
<hashar> |
jenkins: upgrading plugins and restarting. |
[production] |
| 09:43 |
<hashar> |
restarted jenkins twice to get it to finally serve something :( |
[production] |
| 09:26 |
<hashar> |
killed -9 jenkins :( |
[production] |
| 09:25 |
<hashar> |
restarting jenkins |
[production] |
| 09:17 |
<hashar> |
gallium: dumping java heap of Jenkins using jmap -dump:format=b,file=/root/jenkins-bug48025.jmap -F 24508 |
[production] |
| 08:33 |
<hashar> |
Jenkins hit by the evil bug again {{bug|48025}}. Taking as much trace as possible before restarting it |
[production] |
| 04:46 |
<paravoid> |
swift: pushing new rings, set weight 0 for ms-be5 sdb1, ms-be11 sdh1 (broken disks); balance 999.99, needs further rebalance |
[production] |
| 03:37 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri May 10 03:37:41 UTC 2013 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.22wmf3) at Fri May 10 02:15:49 UTC 2013 |
[production] |
| 00:14 |
<pgehres> |
synchronized wmf-config 'Actually enabling wgCentralAuthAutoMigrate' |
[production] |
|
2013-05-09
§
|
| 23:34 |
<mwalker> |
synchronized wmf-config/InitialiseSettings.php 'Enabling siteNotice div on all alpha and beta mobile sites' |
[production] |
| 23:26 |
<pgehres> |
synchronized php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php 'Moving one more CentralAuth log line' |
[production] |
| 23:21 |
<mwalker> |
synchronized php-1.22wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master for some bugfixes' |
[production] |
| 23:06 |
<pgehres> |
synchronized php-1.22wmf3/extensions/CentralAuth/ 'Updates to CentralAuth' |
[production] |
| 21:54 |
<hashar> |
Jenkins / Zuul is back up (for now). |
[production] |
| 21:44 |
<hashar> |
restarted Jenkins to enable access logs in /var/log/jenkins/access.log {{gerrit|63068}} |
[production] |
| 21:42 |
<hashar> |
forced Zuul to stop. All events have been dropped :( |
[production] |
| 21:31 |
<hashar> |
restarting Zuul to make it query Jenkins directly without SSL + apache proxy {{gerrit|63069}}. |
[production] |
| 21:29 |
<Jeff_Green> |
pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed |
[production] |
| 21:28 |
<^demon> |
gerrit repositories will now recursively merge |
[production] |
| 21:16 |
<olivneh> |
synchronized php-1.22wmf3/extensions/WikimediaMessages 'Updating WikimediaMessages to master' |
[production] |
| 20:47 |
<olivneh> |
synchronized php-1.22wmf3/extensions/GuidedTour 'Actually updating GuidedTour to master' |
[production] |
| 20:46 |
<olivneh> |
synchronized php-1.22wmf3/extensions/EventLogging 'Updating GuidedTour to master' |
[production] |
| 20:45 |
<olivneh> |
synchronized php-1.22wmf3/extensions/EventLogging 'Updating EventLogging to master' |
[production] |
| 20:45 |
<olivneh> |
synchronized php-1.22wmf3/extensions/ConfirmEdit 'Updating ConfirmEdit to master' |
[production] |
| 20:20 |
<hashar> |
restarting Jenkins. |
[production] |
| 20:20 |
<hashar> |
Jenkins died :/ |
[production] |
| 19:16 |
<mutante> |
fixing file permissions in /a/e3/E3Analysis repo on stat1001 |
[production] |
| 19:09 |
<mutante> |
restarted Apache and then puppetmaster on stafford, that revived it |
[production] |
| 18:55 |
<mutante> |
attempting to restart broken puppetmaster on stafford |
[production] |
| 18:27 |
<pgehres> |
synchronized wmf-config/InitialiseSettings.php 'Enabling CentralAuth debug log' |
[production] |
| 18:00 |
<pgehres> |
synchronized wmf-config/InitialiseSettings.php 'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth' |
[production] |
| 15:56 |
<cmjohnson1> |
powering down barium /relocate server to c1 |
[production] |
| 15:26 |
<anomie> |
synchronized wmf-config/CommonSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' |
[production] |