|
2010-12-17
§
|
| 03:20 |
<RobH> |
srv240-srv246 online, pushing back into memcached rotation |
[production] |
| 02:30 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv233-srv239 in, srv240-srv246 out for relocation' |
[production] |
| 02:27 |
<RobH> |
removing srv240-srv246 from mc rotation and shutting down for relocation |
[production] |
| 02:27 |
<RobH> |
srv233-srv239 successfully relocated, pushing them into mc rotation |
[production] |
| 01:18 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv233-srv239 moved out of memcached rotation' |
[production] |
| 01:14 |
<RobH> |
srv233-srv239 being removed from memcached pool and shutdown for rack relocation. srv228-srv232 being put back into memcached |
[production] |
| 01:06 |
<RobH> |
srv232 moved and powering up |
[production] |
| 00:46 |
<Ryan_Lane> |
rsyncing data from storage3 to db10 |
[production] |
| 00:43 |
<RobH> |
srv226-srv231 powering up in their new home in d2-sdtpa |
[production] |
| 00:31 |
<Ryan_Lane> |
stopping mysql on storage3 |
[production] |
| 00:31 |
<Ryan_Lane> |
update puppet config for storage3 to update ganglia and nagios |
[production] |
| 00:25 |
<tfinc> |
synchronized php-1.5/wmf-config/reporting-setup.php |
[production] |
| 00:15 |
<Ryan_Lane> |
copying snapshot data to /a/sqldata (excluding faulkner db) |
[production] |
| 00:11 |
<Ryan_Lane> |
deleting databases on db10 |
[production] |
| 00:10 |
<Ryan_Lane> |
stopping mysql on db10 and restoring data from snapshot |
[production] |
| 00:06 |
<Ryan_Lane> |
starting mysql on db10 |
[production] |
| 00:05 |
<Ryan_Lane> |
deleting faulkner db on db10 |
[production] |
|
2010-12-16
§
|
| 23:48 |
<Ryan_Lane> |
starting mysql on storage3 |
[production] |
| 23:42 |
<Ryan_Lane> |
stopping mysql on storage3 to replace the faulkner db |
[production] |
| 23:33 |
<Ryan_Lane> |
starting mysql on storage3 |
[production] |
| 23:07 |
<RobH> |
srv226-srv232 coming down for relocation |
[production] |
| 23:06 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'moved servers around for relocation' |
[production] |
| 22:38 |
<RobH> |
srv281 reinstalling |
[production] |
| 22:24 |
<RobH> |
srv281 coming down for repair (isnt in memcached active pool) |
[production] |
| 22:11 |
<Ryan_Lane> |
rebooting storage3 |
[production] |
| 22:10 |
<Ryan_Lane> |
restating apparmor on storage3 to reload profiles |
[production] |
| 21:09 |
<RobH> |
all power strips in row c and d sdtpa have been wired for serial |
[production] |
| 20:18 |
<Ryan_Lane> |
stopping mysql on db10 and copying data to storage3 |
[production] |
| 20:11 |
<Ryan_Lane> |
installing lvm on storage3 |
[production] |
| 20:11 |
<tomaszf> |
killing long running civicrm queries on db9 |
[production] |
| 19:59 |
<Ryan_Lane> |
creating LVM configuration using sda |
[production] |
| 19:48 |
<Ryan_Lane> |
reformatting /dev/sda1 as xfs on storage3 |
[production] |
| 19:48 |
<Ryan_Lane> |
unmounting /data on storage3 and mounting as /a |
[production] |
| 19:44 |
<CodeBlock> |
installing puppetmaster and puppet on nova-controller |
[production] |
| 19:35 |
<Ryan_Lane> |
installing mysql-client mysql-server mysql-common on storage3 |
[production] |
|
2010-12-15
§
|
| 22:26 |
<Ryan_Lane> |
restarted slave on db10 |
[production] |
| 22:25 |
<Ryan_Lane> |
killed mysqlcheck process on db10, as it was eating all available cpu |
[production] |
| 22:09 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26222 - Set $wgLogo to local logo on rowiki' |
[production] |
| 21:55 |
<Ryan_Lane> |
restarted slave on db10 |
[production] |
| 21:16 |
<Ryan_Lane> |
restarting mysqld on db10 |
[production] |
| 21:16 |
<Ryan_Lane> |
enabling long query log on db10 |
[production] |
| 20:43 |
<Ryan_Lane> |
stoped and started slave on db10 |
[production] |
| 19:39 |
<RobH> |
pushing dns change to switch dumps to dataset1 from singer (download and all its various incarnations are all pointing at dumps |
[production] |
| 19:17 |
<RobH> |
amaranth accidental reboot due to power removal |
[production] |
| 18:50 |
<RobH> |
msw-b1-sdtpa power cycled due to power cable being moved within the rack |
[production] |
| 18:41 |
<RobH> |
storage3 powering down to have its power distro fixed in rack |
[production] |
| 18:30 |
<RobH> |
bringing down db41/db42, the power in rack B1-sdtpa is not balanced properly. |
[production] |
| 17:52 |
<demon> |
synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r78443' |
[production] |
| 17:45 |
<demon> |
synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'Debugging pywikipedia failure' |
[production] |
| 09:48 |
<domas> |
upgraded stuff on secure to fix ssl warnings |
[production] |