|
2012-01-13
§
|
| 22:40 |
<Ryan_Lane> |
deleted all puppet certificates on all instances |
[production] |
| 22:31 |
<LeslieCarr> |
restarting ganglia1001 |
[production] |
| 21:41 |
<Ryan_Lane> |
changing virt1 to be a cname of virt0 |
[production] |
| 21:17 |
<Ryan_Lane> |
killed pdns on virt1 |
[production] |
| 20:40 |
<Ryan_Lane> |
stopped pdns on virt1 |
[production] |
| 20:32 |
<reedy> |
synchronized php-1.18/extensions/Contest/specials/ '[[rev:108843|r108843]]' |
[production] |
| 20:28 |
<Ryan_Lane> |
changed NS records for wmflabs.org and wmflabs to point to virt0 |
[production] |
| 20:28 |
<Ryan_Lane> |
changed recursor to point wmflabs domain to virt0 |
[production] |
| 20:16 |
<K4-713> |
synchronized payments cluster to [[rev:108833|r108833]] |
[production] |
| 19:30 |
<Ryan_Lane> |
shutting down virt1 to ensure migration was completed |
[production] |
| 19:12 |
<LeslieCarr> |
restarting gmond on cp1043 |
[production] |
| 18:55 |
<LeslieCarr> |
hard powercycling ms1002 |
[production] |
| 18:48 |
<LeslieCarr> |
rebooting ms1002 due to kswapd 100% cpu bug https://bugs.launchpad.net/ubuntu/+bug/721896 |
[production] |
| 16:44 |
<mutante> |
added alswiktionary & alswikibooks to closed.dblist |
[production] |
| 16:43 |
<dzahn> |
synchronized closed.dblist |
[production] |
| 16:33 |
<mutante> |
syncing InitialiseSettings.php after changing as wiki namespace per bz 33507 |
[production] |
| 16:33 |
<dzahn> |
synchronized ./wmf-config/InitialiseSettings.php |
[production] |
| 16:07 |
<mutante> |
updated blog theme and installed a plugin per RT 2271 |
[production] |
| 14:34 |
<mutante> |
srv191 - has now fresh OS, re-issued puppet certs, ran puppet, restart memcached, etc. - all back in monitoring |
[production] |
| 12:59 |
<Guest45399> |
PXE booting srv191, installing OS |
[production] |
| 07:45 |
<Ryan_Lane> |
disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. |
[production] |
| 07:43 |
<Ryan_Lane> |
added a grant for mediawiki in the database to fix labsconsole mediawiki outage |
[production] |
| 07:42 |
<Ryan_Lane> |
fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue |
[production] |
| 03:29 |
<Ryan_Lane> |
switching labsconsole.wikimedia.org address to point to virt0 |
[production] |
| 02:58 |
<Ryan_Lane> |
dns server is up on virt0 |
[production] |
| 02:57 |
<Ryan_Lane> |
switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed |
[production] |
| 02:56 |
<Ryan_Lane> |
switched rabbitmq server in labs to virt0 |
[production] |
| 02:56 |
<Ryan_Lane> |
switched mysql masters for labs to virt0 |
[production] |
| 02:05 |
<LocalisationUpdate> |
completed (1.18) at Fri Jan 13 02:05:29 UTC 2012 |
[production] |
| 01:47 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33469 - Enable rollback function for editor group kawiki' |
[production] |
| 01:39 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33507|bug 33507]] for aswiki' |
[production] |
| 01:09 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia' |
[production] |
| 01:02 |
<reedy> |
synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia (resync)' |
[production] |
| 01:02 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33468 - Email notifications for eswikibooks' |
[production] |
| 00:33 |
<Reedy> |
That was only touch |
[production] |
| 00:33 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 00:32 |
<Reedy> |
srv223 is also out of diskspace |
[production] |
| 00:29 |
<Reedy> |
srv219 is out of diskspace |
[production] |
| 00:28 |
<reedy> |
synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia' |
[production] |
| 00:17 |
<Ryan_Lane> |
stopping puppet on all virt nodes |
[production] |
|
2012-01-12
§
|
| 21:54 |
<Ryan_Lane> |
relabeled port at virt0 |
[production] |
| 21:54 |
<Ryan_Lane> |
moved new virt0 from squid vlan to public-services2 |
[production] |
| 21:43 |
<Ryan_Lane> |
rebuilding mobile2 as virt0 |
[production] |
| 21:43 |
<Ryan_Lane> |
Adding back mgmt info for mobile1, changing mobile2 to virt0 |
[production] |
| 21:11 |
<Ryan_Lane> |
rebuilding mobile1 as virt0 |
[production] |
| 21:08 |
<Ryan_Lane> |
renaming mobile1 to virt0 |
[production] |
| 20:54 |
<binasher> |
installing percona-toolkit on few remaining hardy dbs |
[production] |
| 20:26 |
<cmjohnson1> |
shutting down srv178-189 for decommissioning |
[production] |
| 20:14 |
<binasher> |
granted the "process" priv to nagios@localhost on all production db clusters |
[production] |
| 20:07 |
<reedy> |
synchronized php-1.18/includes/specials/SpecialSearch.php '[[rev:108751|r108751]]' |
[production] |