|
2012-01-13
§
|
| 19:30 |
<Ryan_Lane> |
shutting down virt1 to ensure migration was completed |
[production] |
| 19:12 |
<LeslieCarr> |
restarting gmond on cp1043 |
[production] |
| 18:55 |
<LeslieCarr> |
hard powercycling ms1002 |
[production] |
| 18:48 |
<LeslieCarr> |
rebooting ms1002 due to kswapd 100% cpu bug https://bugs.launchpad.net/ubuntu/+bug/721896 |
[production] |
| 16:44 |
<mutante> |
added alswiktionary & alswikibooks to closed.dblist |
[production] |
| 16:43 |
<dzahn> |
synchronized closed.dblist |
[production] |
| 16:33 |
<mutante> |
syncing InitialiseSettings.php after changing as wiki namespace per bz 33507 |
[production] |
| 16:33 |
<dzahn> |
synchronized ./wmf-config/InitialiseSettings.php |
[production] |
| 16:07 |
<mutante> |
updated blog theme and installed a plugin per RT 2271 |
[production] |
| 14:34 |
<mutante> |
srv191 - has now fresh OS, re-issued puppet certs, ran puppet, restart memcached, etc. - all back in monitoring |
[production] |
| 12:59 |
<Guest45399> |
PXE booting srv191, installing OS |
[production] |
| 07:45 |
<Ryan_Lane> |
disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. |
[production] |
| 07:43 |
<Ryan_Lane> |
added a grant for mediawiki in the database to fix labsconsole mediawiki outage |
[production] |
| 07:42 |
<Ryan_Lane> |
fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue |
[production] |
| 03:29 |
<Ryan_Lane> |
switching labsconsole.wikimedia.org address to point to virt0 |
[production] |
| 02:58 |
<Ryan_Lane> |
dns server is up on virt0 |
[production] |
| 02:57 |
<Ryan_Lane> |
switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed |
[production] |
| 02:56 |
<Ryan_Lane> |
switched rabbitmq server in labs to virt0 |
[production] |
| 02:56 |
<Ryan_Lane> |
switched mysql masters for labs to virt0 |
[production] |
| 02:05 |
<LocalisationUpdate> |
completed (1.18) at Fri Jan 13 02:05:29 UTC 2012 |
[production] |
| 01:47 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33469 - Enable rollback function for editor group kawiki' |
[production] |
| 01:39 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33507|bug 33507]] for aswiki' |
[production] |
| 01:09 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia' |
[production] |
| 01:02 |
<reedy> |
synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia (resync)' |
[production] |
| 01:02 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33468 - Email notifications for eswikibooks' |
[production] |
| 00:33 |
<Reedy> |
That was only touch |
[production] |
| 00:33 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 00:32 |
<Reedy> |
srv223 is also out of diskspace |
[production] |
| 00:29 |
<Reedy> |
srv219 is out of diskspace |
[production] |
| 00:28 |
<reedy> |
synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia' |
[production] |
| 00:17 |
<Ryan_Lane> |
stopping puppet on all virt nodes |
[production] |
|
2012-01-12
§
|
| 21:54 |
<Ryan_Lane> |
relabeled port at virt0 |
[production] |
| 21:54 |
<Ryan_Lane> |
moved new virt0 from squid vlan to public-services2 |
[production] |
| 21:43 |
<Ryan_Lane> |
rebuilding mobile2 as virt0 |
[production] |
| 21:43 |
<Ryan_Lane> |
Adding back mgmt info for mobile1, changing mobile2 to virt0 |
[production] |
| 21:11 |
<Ryan_Lane> |
rebuilding mobile1 as virt0 |
[production] |
| 21:08 |
<Ryan_Lane> |
renaming mobile1 to virt0 |
[production] |
| 20:54 |
<binasher> |
installing percona-toolkit on few remaining hardy dbs |
[production] |
| 20:26 |
<cmjohnson1> |
shutting down srv178-189 for decommissioning |
[production] |
| 20:14 |
<binasher> |
granted the "process" priv to nagios@localhost on all production db clusters |
[production] |
| 20:07 |
<reedy> |
synchronized php-1.18/includes/specials/SpecialSearch.php '[[rev:108751|r108751]]' |
[production] |
| 20:07 |
<LeslieCarr> |
reassigning ports on asw-b-sdtpa |
[production] |
| 17:00 |
<notpeter> |
stop sodium to do manual reinstall |
[production] |
| 16:33 |
<RobH> |
adjusting all power strip humidity sensor 2 (floor level) to 12% humidity, as the center rack has the proper levels, floor levels always are low in humidity. |
[production] |
| 16:04 |
<mutante> |
starting nagios-nrpe-server on ALL via dsh to speed up nagios recovery |
[production] |
| 15:33 |
<mutante> |
starting nagios-nrpe-server on srv's via dsh |
[production] |
| 02:04 |
<LocalisationUpdate> |
completed (1.18) at Thu Jan 12 02:04:31 UTC 2012 |
[production] |
| 00:48 |
<preilly> |
pushing quick fix for special random |
[production] |
| 00:48 |
<preilly> |
synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend to fix random link' |
[production] |
| 00:41 |
<LeslieCarr> |
added ganglia1002 and ganglia1001 to dns |
[production] |