|
2010-12-18
ยง
|
| 23:11 |
<RobH> |
srv187-srv189 relocated and powering up |
[production] |
| 22:27 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv181-srv186 in' |
[production] |
| 22:26 |
<mark> |
Updated smokeping with new rack layouts and contents |
[production] |
| 22:21 |
<RobH> |
srv181-srv186 relocated and coming back online |
[production] |
| 22:21 |
<RobH> |
srv187-srv189 coming down for relocation |
[production] |
| 22:00 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv153 srv154 in' |
[production] |
| 21:58 |
<RobH> |
srv153, srv154 powering back up |
[production] |
| 21:53 |
<RobH> |
shutting down srv153, srv154 power redistribution |
[production] |
| 21:51 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv154-srv153 out' |
[production] |
| 21:28 |
<RobH> |
srv181-srv186 going offline for relocation |
[production] |
| 21:27 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv181-srv186 pulled when applicable' |
[production] |
| 21:23 |
<hashar> |
synchronized php-1.5/wmf-config/abusefilter.php 'resyncing abusefilter.php for the offline apaches' |
[production] |
| 21:20 |
<robh> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 21:15 |
<RobH> |
srv157 accidental reboot, opps =P |
[production] |
| 21:13 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'commented out srv156, its coming down for power rebalancing' |
[production] |
| 21:11 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv178-srv180 back in' |
[production] |
| 21:05 |
<RobH> |
srv175-srv180 relocated to new rack and coming online |
[production] |
| 20:53 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv177 back in' |
[production] |
| 20:42 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'srv175 srv176 back in' |
[production] |
| 20:06 |
<hashar> |
synchronized php-1.5/wmf-config/abusefilter.php 'bug 26364: abuse filter privacy issue on hiwiki' |
[production] |
| 20:02 |
<RobH> |
srv175-srv180 coming down for relocation |
[production] |
| 20:02 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'putting ms2 back into ES rotation, removing srv175-srv180' |
[production] |
| 19:39 |
<hashar> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26364: abuse filter privacy issue on hiwiki' |
[production] |
| 19:28 |
<RobH> |
hume back online in new rack |
[production] |
| 18:38 |
<mark> |
DRBD failover to nfs2 successful, nfs2 is now primary |
[production] |
| 18:38 |
<RobH> |
pdf2 relocated, powering up |
[production] |
| 18:32 |
<mark> |
Starting DRBD failover of nfs1 to nfs2 |
[production] |
| 18:20 |
<RobH> |
nfs2 reracked, powered up, online (replication check delayed until later) |
[production] |
| 18:19 |
<RobH> |
pdf2 coming down for relocation, will be back online shortly |
[production] |
| 17:51 |
<RobH> |
shutting down nfs2 for relocation |
[production] |
| 17:42 |
<RobH> |
tridge moved, powering back up |
[production] |
| 17:00 |
<RobH> |
streber & williams moved, powering up |
[production] |
| 16:43 |
<RobH> |
williams and streber coming down for relocation (otrs and rt will be offline during this transition) |
[production] |
| 16:33 |
<RobH> |
mchenry & sanger moved, powering up |
[production] |
| 16:09 |
<RobH> |
shutting down mchenry & sanger to relocate them |
[production] |
| 16:03 |
<mark> |
Started slave on ms1 |
[production] |
| 15:59 |
<mark> |
Renamed asw-b2-pmtpa to asw-d1-sdtpa in DNS |
[production] |
| 15:30 |
<robh> |
synchronized php-1.5/wmf-config/db.php 'remvoing ms5 for relocation, will add back again shortly' |
[production] |
| 15:28 |
<RobH> |
ms1 is going to power down and be relocated |
[production] |
| 15:26 |
<mark> |
Failed over traffic back from amslvs4 to amslvs2 |
[production] |
| 15:23 |
<RobH> |
ms5 moved and powering up |
[production] |
| 15:20 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs2 |
[production] |
| 15:11 |
<mark> |
Failed over upload.esams traffic from amslvs2 to amslvs4 |
[production] |
| 15:04 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs4 |
[production] |
| 14:51 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs3 |
[production] |
| 14:48 |
<mark> |
Failed over traffic back from amslvs3 to amslvs1 |
[production] |
| 14:46 |
<RobH> |
ms5 shutting down for relocation |
[production] |
| 14:32 |
<mark> |
Running apt-get dist-upgrade && reboot on amslvs1 |
[production] |
| 14:27 |
<mark> |
Made puppet install kernel 2.6.36 on LVS balancers |
[production] |
| 14:22 |
<mark> |
ES clusters 7, 20, 21 copies finished |
[production] |