|
2012-01-13
§
|
| 07:45 |
<Ryan_Lane> |
disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. |
[production] |
| 07:43 |
<Ryan_Lane> |
added a grant for mediawiki in the database to fix labsconsole mediawiki outage |
[production] |
| 07:42 |
<Ryan_Lane> |
fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue |
[production] |
| 03:29 |
<Ryan_Lane> |
switching labsconsole.wikimedia.org address to point to virt0 |
[production] |
| 02:58 |
<Ryan_Lane> |
dns server is up on virt0 |
[production] |
| 02:57 |
<Ryan_Lane> |
switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed |
[production] |
| 02:56 |
<Ryan_Lane> |
switched rabbitmq server in labs to virt0 |
[production] |
| 02:56 |
<Ryan_Lane> |
switched mysql masters for labs to virt0 |
[production] |
| 02:05 |
<LocalisationUpdate> |
completed (1.18) at Fri Jan 13 02:05:29 UTC 2012 |
[production] |
| 01:47 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33469 - Enable rollback function for editor group kawiki' |
[production] |
| 01:39 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:33507|bug 33507]] for aswiki' |
[production] |
| 01:09 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia' |
[production] |
| 01:02 |
<reedy> |
synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia (resync)' |
[production] |
| 01:02 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 33468 - Email notifications for eswikibooks' |
[production] |
| 00:33 |
<Reedy> |
That was only touch |
[production] |
| 00:33 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 00:32 |
<Reedy> |
srv223 is also out of diskspace |
[production] |
| 00:29 |
<Reedy> |
srv219 is out of diskspace |
[production] |
| 00:28 |
<reedy> |
synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia' |
[production] |
| 00:17 |
<Ryan_Lane> |
stopping puppet on all virt nodes |
[production] |
|
2012-01-12
§
|
| 21:54 |
<Ryan_Lane> |
relabeled port at virt0 |
[production] |
| 21:54 |
<Ryan_Lane> |
moved new virt0 from squid vlan to public-services2 |
[production] |
| 21:43 |
<Ryan_Lane> |
rebuilding mobile2 as virt0 |
[production] |
| 21:43 |
<Ryan_Lane> |
Adding back mgmt info for mobile1, changing mobile2 to virt0 |
[production] |
| 21:11 |
<Ryan_Lane> |
rebuilding mobile1 as virt0 |
[production] |
| 21:08 |
<Ryan_Lane> |
renaming mobile1 to virt0 |
[production] |
| 20:54 |
<binasher> |
installing percona-toolkit on few remaining hardy dbs |
[production] |
| 20:26 |
<cmjohnson1> |
shutting down srv178-189 for decommissioning |
[production] |
| 20:14 |
<binasher> |
granted the "process" priv to nagios@localhost on all production db clusters |
[production] |
| 20:07 |
<reedy> |
synchronized php-1.18/includes/specials/SpecialSearch.php '[[rev:108751|r108751]]' |
[production] |
| 20:07 |
<LeslieCarr> |
reassigning ports on asw-b-sdtpa |
[production] |
| 17:00 |
<notpeter> |
stop sodium to do manual reinstall |
[production] |
| 16:33 |
<RobH> |
adjusting all power strip humidity sensor 2 (floor level) to 12% humidity, as the center rack has the proper levels, floor levels always are low in humidity. |
[production] |
| 16:04 |
<mutante> |
starting nagios-nrpe-server on ALL via dsh to speed up nagios recovery |
[production] |
| 15:33 |
<mutante> |
starting nagios-nrpe-server on srv's via dsh |
[production] |
| 02:04 |
<LocalisationUpdate> |
completed (1.18) at Thu Jan 12 02:04:31 UTC 2012 |
[production] |
| 00:48 |
<preilly> |
pushing quick fix for special random |
[production] |
| 00:48 |
<preilly> |
synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend to fix random link' |
[production] |
| 00:41 |
<LeslieCarr> |
added ganglia1002 and ganglia1001 to dns |
[production] |
|
2012-01-11
§
|
| 23:18 |
<RobH> |
searchidx1001 offline and powered down until replacement memory arrives (2012-01-13) rt 2208 |
[production] |
| 22:56 |
<RobH> |
poking searchidx1001 for memory error |
[production] |
| 22:45 |
<RobH> |
mw1108 online and ready for install per rt2253 |
[production] |
| 22:42 |
<RobH> |
mw1099 repaired, ready for os install per rt2252 |
[production] |
| 22:39 |
<RobH> |
mw1081 ready for install rt2251 |
[production] |
| 22:32 |
<RobH> |
no its not ;] |
[production] |
| 22:16 |
<Reedy> |
lists.wikimedia.org is down |
[production] |
| 21:53 |
<reedy> |
synchronized php-1.18/includes/api/ '[[rev:108683|r108683]]' |
[production] |
| 21:36 |
<RobH> |
psw1-eqiad mgmt connected |
[production] |
| 21:24 |
<RobH> |
leslie is handling the ganglia not starting back up issue even though i caused it to die, yay me |
[production] |
| 21:22 |
<RobH> |
updated dns for neon/cobalt to ganglia1001/1002 |
[production] |