|
2010-12-17
ยง
|
| 23:38 |
<RobH> |
srv169 is not racked, has a dead hard disk, and had to use its rail for another server until we drill the stuck rail out of old rack |
[production] |
| 23:38 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'pushing srv170-174 back into file' |
[production] |
| 23:36 |
<RobH> |
srv170-srv174 racked and powered, mysql running, putting back into db.php, puppet currently running to bring apache online |
[production] |
| 23:25 |
<apergos> |
added default values for tick and freq to /etc/default/adjtimex on dataset1 manually (ubuntu install of package is broken and leaves broken conf file, known bug, etc.) |
[production] |
| 23:12 |
<mark> |
ES cluster 6 copy done |
[production] |
| 23:00 |
<RobH> |
shutting down and moving srv169-srv174 |
[production] |
| 22:59 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv169-srv174 out for relocation' |
[production] |
| 22:46 |
<mark> |
Started copy of ES cluster 21 data from srv161 to tridge (screen on tridge) |
[production] |
| 22:43 |
<mark> |
Started copy of ES cluster 20 data from srv160 to tridge (screen on tridge) |
[production] |
| 22:41 |
<mark> |
ES cluster 10 copy done |
[production] |
| 22:40 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv157 back in es rotation' |
[production] |
| 22:39 |
<mark> |
Started copy of ES cluster 8 data from srv156 to tridge (screen on tridge) |
[production] |
| 22:38 |
<mark> |
ES cluster 5 copy done |
[production] |
| 22:31 |
<mark> |
Stopped apache & puppet on srv170 to speed up last bit of the copy |
[production] |
| 22:26 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'removing srv157 for relocation' |
[production] |
| 22:25 |
<RobH> |
shutting down srv157 for relocation |
[production] |
| 22:25 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv163-srv168 back into rotation' |
[production] |
| 22:22 |
<RobH> |
srv163-168 relocated, puppet will run automatically to repool apache, mysql is already running, pushing them back into ES cluster |
[production] |
| 22:18 |
<mark> |
ES cluster 9 copy done |
[production] |
| 21:59 |
<mark> |
Stopped puppet and apache on srv157 to speed up copy |
[production] |
| 21:33 |
<RobH> |
shutting down srv163-srv168 for relocation |
[production] |
| 21:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv163-srv168 depooled for relocation' |
[production] |
| 21:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv158-srv162 back into service' |
[production] |
| 21:27 |
<RobH> |
srv158-srv162 back in service, pushing back into db.php |
[production] |
| 21:26 |
<mark> |
ES cluster 4 copy finished |
[production] |
| 21:26 |
<mark> |
ES cluster 3 copy finished |
[production] |
| 20:35 |
<RobH> |
shutting down srv158-srv162, will relocate srv157 after the ES data is backed up to tridge |
[production] |
| 20:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'pulling srv158-srv162 from ES rotation for relocation' |
[production] |
| 20:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv182 is online pushing it back into es service' |
[production] |
| 20:28 |
<mark> |
Started copy of ES cluster 10 data from srv170 to tridge (screen on tridge) |
[production] |
| 20:23 |
<mark> |
Started copy of ES cluster 9 data from srv157 to tridge (screen on tridge) |
[production] |
| 20:16 |
<RobH> |
working on srv169 |
[production] |
| 20:16 |
<mark> |
Started copy of ES cluster 7 data from srv155 to tridge (screen on tridge) |
[production] |
| 20:05 |
<mark> |
Powercycled sq69 |
[production] |
| 19:45 |
<mark> |
Started copy of ES cluster 6 data from srv154 to tridge (screen on tridge) |
[production] |
| 19:37 |
<mark> |
Started copy of ES cluster 5 data from srv153 to tridge (screen on tridge) |
[production] |
| 19:37 |
<mark> |
Started copy of ES cluster 4 data from srv152 to tridge (screen on tridge) |
[production] |
| 19:28 |
<mark> |
Started copy of ES cluster 3 data from srv151 to tridge (screen on tridge) |
[production] |
| 19:18 |
<RobH> |
srv151-srv156 back in service |
[production] |
| 19:17 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'put srv156 back into service' |
[production] |
| 18:39 |
<awjrichards> |
synchronized php-1.5/wmf-config/reporting-setup.php 'Updating maximum amount for contribution reporting' |
[production] |
| 18:31 |
<mark> |
Setup all new PDUs |
[production] |
| 18:17 |
<RobH> |
shutting down srv151-srv156 for relocation |
[production] |
| 17:36 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'moved srv254-srv257 into spares group' |
[production] |
| 17:33 |
<RobH> |
srv254-srv257 back in rotation |
[production] |
| 16:50 |
<mark> |
Set up PDUs ps1-d2-sdtpa and ps1-d3-sdtpa |
[production] |
| 16:39 |
<mark> |
Updated Torrus power monitoring with the new PDUs |
[production] |
| 16:38 |
<RobH> |
srv254-srv257 going offline for relocation |
[production] |
| 16:38 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv254-srv257 replaced in active rotation for relocation' |
[production] |
| 16:03 |
<mark> |
Ignored replication of db bugzilla3 on storage3, repaired bugzilla3 tables, and restarted replication |
[production] |