|
2010-12-17
ยง
|
| 23:00 |
<RobH> |
shutting down and moving srv169-srv174 |
[production] |
| 22:59 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv169-srv174 out for relocation' |
[production] |
| 22:46 |
<mark> |
Started copy of ES cluster 21 data from srv161 to tridge (screen on tridge) |
[production] |
| 22:43 |
<mark> |
Started copy of ES cluster 20 data from srv160 to tridge (screen on tridge) |
[production] |
| 22:41 |
<mark> |
ES cluster 10 copy done |
[production] |
| 22:40 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv157 back in es rotation' |
[production] |
| 22:39 |
<mark> |
Started copy of ES cluster 8 data from srv156 to tridge (screen on tridge) |
[production] |
| 22:38 |
<mark> |
ES cluster 5 copy done |
[production] |
| 22:31 |
<mark> |
Stopped apache & puppet on srv170 to speed up last bit of the copy |
[production] |
| 22:26 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'removing srv157 for relocation' |
[production] |
| 22:25 |
<RobH> |
shutting down srv157 for relocation |
[production] |
| 22:25 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv163-srv168 back into rotation' |
[production] |
| 22:22 |
<RobH> |
srv163-168 relocated, puppet will run automatically to repool apache, mysql is already running, pushing them back into ES cluster |
[production] |
| 22:18 |
<mark> |
ES cluster 9 copy done |
[production] |
| 21:59 |
<mark> |
Stopped puppet and apache on srv157 to speed up copy |
[production] |
| 21:33 |
<RobH> |
shutting down srv163-srv168 for relocation |
[production] |
| 21:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv163-srv168 depooled for relocation' |
[production] |
| 21:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv158-srv162 back into service' |
[production] |
| 21:27 |
<RobH> |
srv158-srv162 back in service, pushing back into db.php |
[production] |
| 21:26 |
<mark> |
ES cluster 4 copy finished |
[production] |
| 21:26 |
<mark> |
ES cluster 3 copy finished |
[production] |
| 20:35 |
<RobH> |
shutting down srv158-srv162, will relocate srv157 after the ES data is backed up to tridge |
[production] |
| 20:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'pulling srv158-srv162 from ES rotation for relocation' |
[production] |
| 20:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv182 is online pushing it back into es service' |
[production] |
| 20:28 |
<mark> |
Started copy of ES cluster 10 data from srv170 to tridge (screen on tridge) |
[production] |
| 20:23 |
<mark> |
Started copy of ES cluster 9 data from srv157 to tridge (screen on tridge) |
[production] |
| 20:16 |
<RobH> |
working on srv169 |
[production] |
| 20:16 |
<mark> |
Started copy of ES cluster 7 data from srv155 to tridge (screen on tridge) |
[production] |
| 20:05 |
<mark> |
Powercycled sq69 |
[production] |
| 19:45 |
<mark> |
Started copy of ES cluster 6 data from srv154 to tridge (screen on tridge) |
[production] |
| 19:37 |
<mark> |
Started copy of ES cluster 5 data from srv153 to tridge (screen on tridge) |
[production] |
| 19:37 |
<mark> |
Started copy of ES cluster 4 data from srv152 to tridge (screen on tridge) |
[production] |
| 19:28 |
<mark> |
Started copy of ES cluster 3 data from srv151 to tridge (screen on tridge) |
[production] |
| 19:18 |
<RobH> |
srv151-srv156 back in service |
[production] |
| 19:17 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'put srv156 back into service' |
[production] |
| 18:39 |
<awjrichards> |
synchronized php-1.5/wmf-config/reporting-setup.php 'Updating maximum amount for contribution reporting' |
[production] |
| 18:31 |
<mark> |
Setup all new PDUs |
[production] |
| 18:17 |
<RobH> |
shutting down srv151-srv156 for relocation |
[production] |
| 17:36 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'moved srv254-srv257 into spares group' |
[production] |
| 17:33 |
<RobH> |
srv254-srv257 back in rotation |
[production] |
| 16:50 |
<mark> |
Set up PDUs ps1-d2-sdtpa and ps1-d3-sdtpa |
[production] |
| 16:39 |
<mark> |
Updated Torrus power monitoring with the new PDUs |
[production] |
| 16:38 |
<RobH> |
srv254-srv257 going offline for relocation |
[production] |
| 16:38 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv254-srv257 replaced in active rotation for relocation' |
[production] |
| 16:03 |
<mark> |
Ignored replication of db bugzilla3 on storage3, repaired bugzilla3 tables, and restarted replication |
[production] |
| 15:55 |
<mark> |
Setup DNS for new PDUs |
[production] |
| 15:16 |
<mark> |
Updating dns for asw-b1-pmtpa and asw-b4-pmtpa name change |
[production] |
| 04:08 |
<RobH> |
srv247-srv253 back in rotation and relocated into d2 from b2 (will update racktables shortly) |
[production] |
| 03:26 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv240-srv246 in srv247-srv253 out' |
[production] |
| 03:21 |
<RobH> |
srv247-srv253 removing from memcached rotation and shutting down for rack relocation |
[production] |