|
2010-12-17
§
|
| 21:26 |
<mark> |
ES cluster 4 copy finished |
[production] |
| 21:26 |
<mark> |
ES cluster 3 copy finished |
[production] |
| 20:35 |
<RobH> |
shutting down srv158-srv162, will relocate srv157 after the ES data is backed up to tridge |
[production] |
| 20:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'pulling srv158-srv162 from ES rotation for relocation' |
[production] |
| 20:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv182 is online pushing it back into es service' |
[production] |
| 20:28 |
<mark> |
Started copy of ES cluster 10 data from srv170 to tridge (screen on tridge) |
[production] |
| 20:23 |
<mark> |
Started copy of ES cluster 9 data from srv157 to tridge (screen on tridge) |
[production] |
| 20:16 |
<RobH> |
working on srv169 |
[production] |
| 20:16 |
<mark> |
Started copy of ES cluster 7 data from srv155 to tridge (screen on tridge) |
[production] |
| 20:05 |
<mark> |
Powercycled sq69 |
[production] |
| 19:45 |
<mark> |
Started copy of ES cluster 6 data from srv154 to tridge (screen on tridge) |
[production] |
| 19:37 |
<mark> |
Started copy of ES cluster 5 data from srv153 to tridge (screen on tridge) |
[production] |
| 19:37 |
<mark> |
Started copy of ES cluster 4 data from srv152 to tridge (screen on tridge) |
[production] |
| 19:28 |
<mark> |
Started copy of ES cluster 3 data from srv151 to tridge (screen on tridge) |
[production] |
| 19:18 |
<RobH> |
srv151-srv156 back in service |
[production] |
| 19:17 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'put srv156 back into service' |
[production] |
| 18:39 |
<awjrichards> |
synchronized php-1.5/wmf-config/reporting-setup.php 'Updating maximum amount for contribution reporting' |
[production] |
| 18:31 |
<mark> |
Setup all new PDUs |
[production] |
| 18:17 |
<RobH> |
shutting down srv151-srv156 for relocation |
[production] |
| 17:36 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'moved srv254-srv257 into spares group' |
[production] |
| 17:33 |
<RobH> |
srv254-srv257 back in rotation |
[production] |
| 16:50 |
<mark> |
Set up PDUs ps1-d2-sdtpa and ps1-d3-sdtpa |
[production] |
| 16:39 |
<mark> |
Updated Torrus power monitoring with the new PDUs |
[production] |
| 16:38 |
<RobH> |
srv254-srv257 going offline for relocation |
[production] |
| 16:38 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv254-srv257 replaced in active rotation for relocation' |
[production] |
| 16:03 |
<mark> |
Ignored replication of db bugzilla3 on storage3, repaired bugzilla3 tables, and restarted replication |
[production] |
| 15:55 |
<mark> |
Setup DNS for new PDUs |
[production] |
| 15:16 |
<mark> |
Updating dns for asw-b1-pmtpa and asw-b4-pmtpa name change |
[production] |
| 04:08 |
<RobH> |
srv247-srv253 back in rotation and relocated into d2 from b2 (will update racktables shortly) |
[production] |
| 03:26 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv240-srv246 in srv247-srv253 out' |
[production] |
| 03:21 |
<RobH> |
srv247-srv253 removing from memcached rotation and shutting down for rack relocation |
[production] |
| 03:20 |
<RobH> |
srv240-srv246 online, pushing back into memcached rotation |
[production] |
| 02:30 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv233-srv239 in, srv240-srv246 out for relocation' |
[production] |
| 02:27 |
<RobH> |
removing srv240-srv246 from mc rotation and shutting down for relocation |
[production] |
| 02:27 |
<RobH> |
srv233-srv239 successfully relocated, pushing them into mc rotation |
[production] |
| 01:18 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv233-srv239 moved out of memcached rotation' |
[production] |
| 01:14 |
<RobH> |
srv233-srv239 being removed from memcached pool and shutdown for rack relocation. srv228-srv232 being put back into memcached |
[production] |
| 01:06 |
<RobH> |
srv232 moved and powering up |
[production] |
| 00:46 |
<Ryan_Lane> |
rsyncing data from storage3 to db10 |
[production] |
| 00:43 |
<RobH> |
srv226-srv231 powering up in their new home in d2-sdtpa |
[production] |
| 00:31 |
<Ryan_Lane> |
stopping mysql on storage3 |
[production] |
| 00:31 |
<Ryan_Lane> |
update puppet config for storage3 to update ganglia and nagios |
[production] |
| 00:25 |
<tfinc> |
synchronized php-1.5/wmf-config/reporting-setup.php |
[production] |
| 00:15 |
<Ryan_Lane> |
copying snapshot data to /a/sqldata (excluding faulkner db) |
[production] |
| 00:11 |
<Ryan_Lane> |
deleting databases on db10 |
[production] |
| 00:10 |
<Ryan_Lane> |
stopping mysql on db10 and restoring data from snapshot |
[production] |
| 00:06 |
<Ryan_Lane> |
starting mysql on db10 |
[production] |
| 00:05 |
<Ryan_Lane> |
deleting faulkner db on db10 |
[production] |