|
2010-12-18
§
|
| 18:32 |
<mark> |
Starting DRBD failover of nfs1 to nfs2 |
[production] |
| 18:20 |
<RobH> |
nfs2 reracked, powered up, online (replication check delayed until later) |
[production] |
| 18:19 |
<RobH> |
pdf2 coming down for relocation, will be back online shortly |
[production] |
| 17:51 |
<RobH> |
shutting down nfs2 for relocation |
[production] |
| 17:42 |
<RobH> |
tridge moved, powering back up |
[production] |
| 17:00 |
<RobH> |
streber & williams moved, powering up |
[production] |
| 16:43 |
<RobH> |
williams and streber coming down for relocation (otrs and rt will be offline during this transition) |
[production] |
| 16:33 |
<RobH> |
mchenry & sanger moved, powering up |
[production] |
| 16:09 |
<RobH> |
shutting down mchenry & sanger to relocate them |
[production] |
| 16:03 |
<mark> |
Started slave on ms1 |
[production] |
| 15:59 |
<mark> |
Renamed asw-b2-pmtpa to asw-d1-sdtpa in DNS |
[production] |
| 15:30 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'remvoing ms5 for relocation, will add back again shortly' |
[production] |
| 15:28 |
<RobH> |
ms1 is going to power down and be relocated |
[production] |
| 15:26 |
<mark> |
Failed over traffic back from amslvs4 to amslvs2 |
[production] |
| 15:23 |
<RobH> |
ms5 moved and powering up |
[production] |
| 15:20 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs2 |
[production] |
| 15:11 |
<mark> |
Failed over upload.esams traffic from amslvs2 to amslvs4 |
[production] |
| 15:04 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs4 |
[production] |
| 14:51 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs3 |
[production] |
| 14:48 |
<mark> |
Failed over traffic back from amslvs3 to amslvs1 |
[production] |
| 14:46 |
<RobH> |
ms5 shutting down for relocation |
[production] |
| 14:32 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs1 |
[production] |
| 14:27 |
<mark> |
Made puppet install kernel 2.6.36 on LVS balancers |
[production] |
| 14:22 |
<mark> |
ES clusters 7, 20, 21 copies finished |
[production] |
| 14:22 |
<mark> |
ES cluster 8 copy finished |
[production] |
| 14:07 |
<mark> |
Imported package "linux-image-2.6.36-1-server" from the kernel PPA into the Wikimedia APT repository for lucid-wikimedia, section universe |
[production] |
|
2010-12-17
§
|
| 23:38 |
<RobH> |
srv169 is not racked, has a dead hard disk, and had to use its rail for another server until we drill the stuck rail out of old rack |
[production] |
| 23:38 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'pushing srv170-174 back into file' |
[production] |
| 23:36 |
<RobH> |
srv170-srv174 racked and powered, mysql running, putting back into db.php, puppet currently running to bring apache online |
[production] |
| 23:25 |
<apergos> |
added default values for tick and freq to /etc/default/adjtimex on dataset1 manually (ubuntu install of package is broken and leaves broken conf file, known bug, etc.) |
[production] |
| 23:12 |
<mark> |
ES cluster 6 copy done |
[production] |
| 23:00 |
<RobH> |
shutting down and moving srv169-srv174 |
[production] |
| 22:59 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv169-srv174 out for relocation' |
[production] |
| 22:46 |
<mark> |
Started copy of ES cluster 21 data from srv161 to tridge (screen on tridge) |
[production] |
| 22:43 |
<mark> |
Started copy of ES cluster 20 data from srv160 to tridge (screen on tridge) |
[production] |
| 22:41 |
<mark> |
ES cluster 10 copy done |
[production] |
| 22:40 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv157 back in es rotation' |
[production] |
| 22:39 |
<mark> |
Started copy of ES cluster 8 data from srv156 to tridge (screen on tridge) |
[production] |
| 22:38 |
<mark> |
ES cluster 5 copy done |
[production] |
| 22:31 |
<mark> |
Stopped apache & puppet on srv170 to speed up last bit of the copy |
[production] |
| 22:26 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'removing srv157 for relocation' |
[production] |
| 22:25 |
<RobH> |
shutting down srv157 for relocation |
[production] |
| 22:25 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv163-srv168 back into rotation' |
[production] |
| 22:22 |
<RobH> |
srv163-168 relocated, puppet will run automatically to repool apache, mysql is already running, pushing them back into ES cluster |
[production] |
| 22:18 |
<mark> |
ES cluster 9 copy done |
[production] |
| 21:59 |
<mark> |
Stopped puppet and apache on srv157 to speed up copy |
[production] |
| 21:33 |
<RobH> |
shutting down srv163-srv168 for relocation |
[production] |
| 21:32 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv163-srv168 depooled for relocation' |
[production] |
| 21:28 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting srv158-srv162 back into service' |
[production] |
| 21:27 |
<RobH> |
srv158-srv162 back in service, pushing back into db.php |
[production] |