|
2010-12-18
§
|
| 15:28 |
<RobH> |
ms1 is going to power down and be relocated |
[production] |
| 15:26 |
<mark> |
Failed over traffic back from amslvs4 to amslvs2 |
[production] |
| 15:23 |
<RobH> |
ms5 moved and powering up |
[production] |
| 15:20 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs2 |
[production] |
| 15:11 |
<mark> |
Failed over upload.esams traffic from amslvs2 to amslvs4 |
[production] |
| 15:04 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs4 |
[production] |
| 14:51 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs3 |
[production] |
| 14:48 |
<mark> |
Failed over traffic back from amslvs3 to amslvs1 |
[production] |
| 14:46 |
<RobH> |
ms5 shutting down for relocation |
[production] |
| 14:32 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs1 |
[production] |
| 14:27 |
<mark> |
Made puppet install kernel 2.6.36 on LVS balancers |
[production] |
| 14:22 |
<mark> |
ES clusters 7, 20, 21 copies finished |
[production] |
| 14:22 |
<mark> |
ES cluster 8 copy finished |
[production] |
| 14:07 |
<mark> |
Imported package "linux-image-2.6.36-1-server" from the kernel PPA into the Wikimedia APT repository for lucid-wikimedia, section universe |
[production] |
|
2010-12-17
§
|
| 23:38 |
<RobH> |
srv169 is not racked, has a dead hard disk, and had to use its rail for another server until we drill the stuck rail out of old rack |
[production] |
| 23:38 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'pushing srv170-174 back into file' |
[production] |
| 23:36 |
<RobH> |
srv170-srv174 racked and powered, mysql running, putting back into db.php, puppet currently running to bring apache online |
[production] |
| 23:25 |
<apergos> |
added default values for tick and freq to /etc/default/adjtimex on dataset1 manually (ubuntu install of package is broken and leaves broken conf file, known bug, etc.) |
[production] |
| 23:12 |
<mark> |
ES cluster 6 copy done |
[production] |
| 23:00 |
<RobH> |
shutting down and moving srv169-srv174 |
[production] |
| 22:59 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv169-srv174 out for relocation' |
[production] |
| 22:46 |
<mark> |
Started copy of ES cluster 21 data from srv161 to tridge (screen on tridge) |
[production] |
| 22:43 |
<mark> |
Started copy of ES cluster 20 data from srv160 to tridge (screen on tridge) |
[production] |
| 22:41 |
<mark> |
ES cluster 10 copy done |
[production] |
| 22:40 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv157 back in es rotation' |
[production] |
| 22:39 |
<mark> |
Started copy of ES cluster 8 data from srv156 to tridge (screen on tridge) |
[production] |
| 22:38 |
<mark> |
ES cluster 5 copy done |
[production] |
| 22:31 |
<mark> |
Stopped apache & puppet on srv170 to speed up last bit of the copy |
[production] |
| 22:26 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'removing srv157 for relocation' |
[production] |
| 22:25 |
<RobH> |
shutting down srv157 for relocation |
[production] |
| 22:25 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv163-srv168 back into rotation' |
[production] |
| 22:22 |
<RobH> |
srv163-168 relocated, puppet will run automatically to repool apache, mysql is already running, pushing them back into ES cluster |
[production] |
| 22:18 |
<mark> |
ES cluster 9 copy done |
[production] |
| 21:59 |
<mark> |
Stopped puppet and apache on srv157 to speed up copy |
[production] |
| 21:33 |
<RobH> |
shutting down srv163-srv168 for relocation |
[production] |
| 21:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv163-srv168 depooled for relocation' |
[production] |
| 21:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv158-srv162 back into service' |
[production] |
| 21:27 |
<RobH> |
srv158-srv162 back in service, pushing back into db.php |
[production] |
| 21:26 |
<mark> |
ES cluster 4 copy finished |
[production] |
| 21:26 |
<mark> |
ES cluster 3 copy finished |
[production] |
| 20:35 |
<RobH> |
shutting down srv158-srv162, will relocate srv157 after the ES data is backed up to tridge |
[production] |
| 20:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'pulling srv158-srv162 from ES rotation for relocation' |
[production] |
| 20:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv182 is online pushing it back into es service' |
[production] |
| 20:28 |
<mark> |
Started copy of ES cluster 10 data from srv170 to tridge (screen on tridge) |
[production] |
| 20:23 |
<mark> |
Started copy of ES cluster 9 data from srv157 to tridge (screen on tridge) |
[production] |
| 20:16 |
<RobH> |
working on srv169 |
[production] |
| 20:16 |
<mark> |
Started copy of ES cluster 7 data from srv155 to tridge (screen on tridge) |
[production] |
| 20:05 |
<mark> |
Powercycled sq69 |
[production] |
| 19:45 |
<mark> |
Started copy of ES cluster 6 data from srv154 to tridge (screen on tridge) |
[production] |
| 19:37 |
<mark> |
Started copy of ES cluster 5 data from srv153 to tridge (screen on tridge) |
[production] |