751-800 of 6216 results (6ms)
2010-01-26 §
17:52 <Rob> bayes is not coming back online properly, checking. [production]
17:47 <Rob> bayes rack location moved and updated in racktables, powering back up [production]
17:34 <Rob> shutting down bayes to move its rack location from b2-pmtpa to a2-pmtpa [production]
14:42 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22274: Set $wgAutoConfirmCount to 10 on itwiktionary' [production]
00:56 <Tim> on hume: increased RCT proc count, reduced max lag to 1s, increased lag check frequency [production]
2010-01-25 §
22:36 <Tim> ms3 is reporting "table full" for rc1 enwiki.blobs. Running ALTER TABLE. [production]
19:33 <Fred> stopped apache on srv225 [production]
19:01 <Fred> adjusted apache.conf on srv225 to make it work again... [production]
16:07 <Rob> powering srv224 & srv225 back up [production]
16:01 <Rob> shutting down srv225 & srv224 to swap their power cables from single to Y split. [production]
15:45 <Rob> srv196 memory replaced, back online and in service [production]
15:45 <root> synchronized php-1.5/wmf-config/mc.php 'pushing out change to two hosts that were down a few minutes ago' [production]
15:40 <Rob> shutting down db28 to swap memory around in diagnostics (bad memory in system) [production]
15:25 <root> synchronized php-1.5/wmf-config/mc.php 'took down srv196 for hardware replacement' [production]
15:20 <Rob> shutting down srv196 to replace faulty memory dimm [production]
12:19 <Tim> started recompressTracked.sh on hume [production]
2010-01-23 §
00:16 <aZaFred_> implemented ganglia detailled mysql stats for db10. [production]
00:16 <aZaFred_> implemented DB Lag detection script for DB10 [production]
2010-01-22 §
23:08 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.hooks.php 'Deploy r61401' [production]
23:07 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/CollapsibleTabs/CollapsibleTabs.hooks.php 'Deploy r61403' [production]
21:29 <brion> freed some space on srv218 removing old upload temp files from /tmp but / is still pretty tight (445m free) [production]
20:19 <catrope> synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'Deploying r61387' [production]
19:05 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22201: Add WZ: alias on itwiktionary' [production]
15:22 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable $wgEnotifWatchlist on usabilitywiki' [production]
11:38 <mark> Deployed new gmond 3.0.3-2 package that conflicts with the incompatible ganglia 3.1 packages in Ubuntu, to avoid problems in our upcoming upgrade [production]
2010-01-21 §
14:13 <mark> Added srv210 to LVS [production]
14:11 <mark> Reenabled srv245 in LVS [production]
13:58 <mark> Restarted Apache on srv207, srv210, srv233, srv257, srv245 [production]
09:30 <mark> Replaced srv121 (decommissioned) by search13 in smokeping [production]
07:59 <AaronSchulz> Removed empty ct_tag rows from code_tags [production]
07:47 <Tim> cleaned up /tmp on srv180 (was out of disk space) [production]
2010-01-20 §
20:36 <root> synchronized php-1.5/wmf-config/mc.php [production]
16:38 <Fred> killed long running queries created by civicrm on db9. [production]
13:56 <domas> did lots of changes on wikitech wiki. :) [production]
13:55 <domas> test! [production]
02:38 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki' [production]
01:20 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki' [production]
01:16 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling API request logging on dewiki' [production]
00:03 <RoanKattouw> Strike my last, was actually *enabling* it on dewiki [production]
00:03 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on dewiki' [production]
00:02 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on frwiki' [production]
2010-01-19 §
21:37 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling API request logging on frwiki' [production]
21:36 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling API request logging on frwiki' [production]
20:34 <Fred> bugzilla upgrade completed. It might be up to an hour before attachments can be added due to DNS propagation. [production]
19:41 <Fred> Bugzilla as you know it is going down. Be back shortly. [production]
18:21 <Rob> updated blog.wikimedia.org wp-stats plugin to 1.6.1 [production]
16:03 <Rob> ssds installed in db28, during boot it displays a memory error, investigating. [production]
15:20 <Rob> db28 offline while disks are swapped to solid state for testing. [production]
2010-01-18 §
05:02 <Tim> removed some old xff logs to free up space for logrotate which I ran with the -f flag [production]
05:00 <Tim> fixed logrotate on nfs1, broken due to duplicated entries between /etc/logrotate.d/rsyslog and /etc/logrotate.d/syslog-ng. Left the syslog-ng ones in for now. [production]