401-450 of 8252 results (11ms)
2010-08-09 §
22:33 <RobH> knsq23, knsq24 back online, knsq25, knsq26 still being reinstalled, knsq27-30 still online not yet reinstalled [production]
21:51 <mark> Added Nagios router interfaces check for br1-knams (using puppet) [production]
21:11 <mark> Unmounted /dev/sda6 (/a) on srv171, replaced it by /dev/mapper/nonredundant-data (LV with the same data and more space) [production]
21:02 <RobH> knsq24, knsq25, knsq26, knsq27 coming down for reinstall and puppetfication [production]
20:49 <RobH> knsq23 reinstall done and pushed back into cluster [production]
18:40 <mark> Running apt-get upgrade on db9 [production]
18:29 <mark> Fixed ganglia mess on sq45 [production]
18:24 <mark> Powercycled sq45 [production]
18:18 <mark> Added a new MegaCli64 to wikimedia-raid-utils, made check-raid.py use it instead (we have all 64 bit servers anyway), and deployed the new package to the repository. Puppet will upgrade it everywhere. [production]
16:26 <Fred> fixed DPKG issue on transcode... another one of those conflicting gmond install [production]
16:14 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24735: Sanitize private/fishbowl config' [production]
16:03 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24732: Portal and Book namespaces for yowiki' [production]
13:59 <mark> srv110 decommissioned itself [production]
13:55 <RobH> knsq23 coming down for reinstallation [production]
13:31 <mark> Changed broken HTTP nagios check for Squid on brewster into a TCP port check [production]
13:28 <mark> Stopped MySQL on srv171, created LVM PV,VG and LV on unused drive /dev/sdb. Copying MySQL data onto it. [production]
13:09 <mark> START SLAVE on srv171 to get rid of relay binlogs [production]
12:56 <mark> Shutdown db3 for decommissioning [production]
12:56 <RoanKattouw> Mark 12:53 Shutdown db2 for decommissioning [production]
12:56 <RoanKattouw> 12:52 mark synchronized php-1.5/wmf-config/db.php 'Remove db3 from rotation, decommissioning' [production]
12:55 <RoanKattouw> Mark 12:47 Power cycled pdf3, out of memory [production]
12:55 <RoanKattouw> Mark 12:44 Restarted Apache on srv91 [production]
12:55 <RoanKattouw> Mark 12:39 Relaxed NTP peers check for dobson and linne (NTP servers) [production]
12:55 <RoanKattouw> Mark 12:36 Shutdown adler for decommissioning [production]
12:54 <RoanKattouw> Mark 12:18 Made disk space on mchenry by DELETING LOTS OF OLD BACKUPS [production]
12:53 <RoanKattouw> Restarted morebots [production]
2010-08-08 §
20:43 <mark> Setup new aggregated graphs at http://torrus.wikimedia.org/torrus/CDN?path=%2FTotals%2F (including bits) [production]
09:49 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24715: Fix typo' [production]
2010-08-07 §
11:58 <mark> synchronized php-1.5/wmf-config/db.php 'New master: db18, r/w' [production]
11:58 <mark> Changed master for s3 to db18 on db11, db27, db25 [production]
11:49 <mark> New master db18 log position: db18.bin.001 pos 79 [production]
11:49 <mark> New master db18 log position: db-18.bin.001 pos 79 [production]
11:32 <mark> synchronized php-1.5/wmf-config/db.php 'Setting s3 to read-only' [production]
11:21 <mark> Stopping mysql on db17 [production]
11:02 <RoanKattouw> All s3 slaves down, master serving all read load and getting overloaded [production]
11:00 <RoanKattouw> db17 (s3 master) has full disk [production]
2010-08-06 §
22:59 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' [production]
22:59 <catrope> synchronized php-1.5/includes/GlobalFunctions.php 'r70605' [production]
22:58 <catrope> synchronized php-1.5/skins/vector/main-rtl.css 'r70605' [production]
22:56 <catrope> synchronized php-1.5/skins/vector/main-ltr.css 'r70605' [production]
15:43 <catrope> synchronized php-1.5/languages/messages/MessagesCs.php 'r70573' [production]
15:08 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24688: Namespace aliases for kowiki' [production]
10:45 <catrope> synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r70550' [production]
10:11 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add foundationwiki addgroups in correct section' [production]
2010-08-05 §
21:14 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24678: Set $wgAddGroups, $wgRemoveGroups on foundationwiki' [production]
14:30 <RobH> replacing certificate file on sanger [production]
2010-08-04 §
22:57 <rainman-sr> search1 somehow got stuck, restarting [production]
22:42 <Fred> restarted Nagios bot [production]
16:28 <mark> Reverted Fred's automatic security upgrades in puppet [production]
16:25 <mark> base::puppet and base::apt were not being included on every Linux host, fixed the case statement in the Puppet base class [production]