|
2010-08-09
§
|
| 23:34 |
<RobH> |
knsq28 reinstalling |
[production] |
| 23:32 |
<RobH> |
knsq26 online |
[production] |
| 23:32 |
<RobH> |
knsq25 online |
[production] |
| 23:12 |
<RobH> |
continuing reinstallation, ignore errors for knsq27, reinstalling |
[production] |
| 22:33 |
<RobH> |
knsq23, knsq24 back online, knsq25, knsq26 still being reinstalled, knsq27-30 still online not yet reinstalled |
[production] |
| 21:51 |
<mark> |
Added Nagios router interfaces check for br1-knams (using puppet) |
[production] |
| 21:11 |
<mark> |
Unmounted /dev/sda6 (/a) on srv171, replaced it by /dev/mapper/nonredundant-data (LV with the same data and more space) |
[production] |
| 21:02 |
<RobH> |
knsq24, knsq25, knsq26, knsq27 coming down for reinstall and puppetfication |
[production] |
| 20:49 |
<RobH> |
knsq23 reinstall done and pushed back into cluster |
[production] |
| 18:40 |
<mark> |
Running apt-get upgrade on db9 |
[production] |
| 18:29 |
<mark> |
Fixed ganglia mess on sq45 |
[production] |
| 18:24 |
<mark> |
Powercycled sq45 |
[production] |
| 18:18 |
<mark> |
Added a new MegaCli64 to wikimedia-raid-utils, made check-raid.py use it instead (we have all 64 bit servers anyway), and deployed the new package to the repository. Puppet will upgrade it everywhere. |
[production] |
| 16:26 |
<Fred> |
fixed DPKG issue on transcode... another one of those conflicting gmond install |
[production] |
| 16:14 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24735: Sanitize private/fishbowl config' |
[production] |
| 16:03 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24732: Portal and Book namespaces for yowiki' |
[production] |
| 13:59 |
<mark> |
srv110 decommissioned itself |
[production] |
| 13:55 |
<RobH> |
knsq23 coming down for reinstallation |
[production] |
| 13:31 |
<mark> |
Changed broken HTTP nagios check for Squid on brewster into a TCP port check |
[production] |
| 13:28 |
<mark> |
Stopped MySQL on srv171, created LVM PV,VG and LV on unused drive /dev/sdb. Copying MySQL data onto it. |
[production] |
| 13:09 |
<mark> |
START SLAVE on srv171 to get rid of relay binlogs |
[production] |
| 12:56 |
<mark> |
Shutdown db3 for decommissioning |
[production] |
| 12:56 |
<RoanKattouw> |
Mark 12:53 Shutdown db2 for decommissioning |
[production] |
| 12:56 |
<RoanKattouw> |
12:52 mark synchronized php-1.5/wmf-config/db.php 'Remove db3 from rotation, decommissioning' |
[production] |
| 12:55 |
<RoanKattouw> |
Mark 12:47 Power cycled pdf3, out of memory |
[production] |
| 12:55 |
<RoanKattouw> |
Mark 12:44 Restarted Apache on srv91 |
[production] |
| 12:55 |
<RoanKattouw> |
Mark 12:39 Relaxed NTP peers check for dobson and linne (NTP servers) |
[production] |
| 12:55 |
<RoanKattouw> |
Mark 12:36 Shutdown adler for decommissioning |
[production] |
| 12:54 |
<RoanKattouw> |
Mark 12:18 Made disk space on mchenry by DELETING LOTS OF OLD BACKUPS |
[production] |
| 12:53 |
<RoanKattouw> |
Restarted morebots |
[production] |
|
2010-08-07
§
|
| 11:58 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'New master: db18, r/w' |
[production] |
| 11:58 |
<mark> |
Changed master for s3 to db18 on db11, db27, db25 |
[production] |
| 11:49 |
<mark> |
New master db18 log position: db18.bin.001 pos 79 |
[production] |
| 11:49 |
<mark> |
New master db18 log position: db-18.bin.001 pos 79 |
[production] |
| 11:32 |
<mark> |
synchronized php-1.5/wmf-config/db.php 'Setting s3 to read-only' |
[production] |
| 11:21 |
<mark> |
Stopping mysql on db17 |
[production] |
| 11:02 |
<RoanKattouw> |
All s3 slaves down, master serving all read load and getting overloaded |
[production] |
| 11:00 |
<RoanKattouw> |
db17 (s3 master) has full disk |
[production] |