651-700 of 4227 results (7ms)
2009-09-15 §
15:59 <Rob> rebooting srv245 to fix its drac access [production]
15:33 <Rob> some odd invalid entries in dsh nodes, removed. [production]
15:24 <Rob> running wipe on srv35, srv52, srv54, srv56 [production]
15:20 <Rob> srv66 running wipe [production]
15:12 <mark> Started MySQL on ms2 and restarted replication [production]
15:11 <mark> Redistributed the spare drives on ms2 back into the spare pool (/dev/md1) [production]
15:08 <Rob> shutting down db12 for raid battery swap [production]
15:04 <mark> Swapped drive c3t6d0 in ms2, readded it to /dev/md14 and moved back the spare /dev/sdao into the spare pool (/dev/md1) [production]
14:42 <mark> Shutting down MySQL on ms2 [production]
14:33 <Rob> removed a number of decommissioned servers from nagios [production]
14:30 <Rob> wipe running on srv44, srv45, srv47 [production]
14:26 <Rob> srv31, srv32, srv33 running wipe in screen sessions, do not try to use them ;] [production]
14:22 <Rob> srv30=srv80 to be decommissioned, wiping the drives with them in rack now. mark already depooled from apache and memcached [production]
14:19 <Rob> srv145 is back up and ok [production]
14:07 <Rob> srv145 coming back up, my bad [production]
13:53 <Rob> srv52 hdd died. [production]
13:51 <domas> cron jobs work way better, if one figures how to set permissions right (like, executable? :) [production]
00:03 <atglenn> grrr.. that would be ms4. [production]
00:03 <atglenn> testing mailiferr (workaround for no MAILTO on solaris) on ms5 for hourly snaps [production]
2009-09-14 §
23:26 <atglenn> rerunning the rsync list of changed files again on ms6 (last run was borked). it's in screen as root [production]
23:23 <aZaFred_> added nagios/ganglia monitoring for snapshot[1..3] [production]
22:32 <atglenn> created a directory /export/upload/wikipedia/common/thumb with no write perms on ms1 so that static html dump ext doesn't try to write in it [production]
21:39 <aZaFred_> upgrading Ubuntu on Sage to 8.04 LTS [production]
21:12 <tomaszf> running timings for en static html snapshot from hume [production]
19:50 <robh> synchronized php-1.5/wmf-config/InitialiseSettings.php [production]
19:47 <Rob> left out the virtual host, resyncing and restarting apaches [production]
19:44 <robh> ran sync-common-all [production]
18:21 <Rob> added volunteer.wikimedia.org to dns and pushed authdns-update [production]
18:03 <Rob> added flagrevs stats update to hume crontab [production]
16:24 <brion> restarted apache on wikitech [production]
2009-09-13 §
21:03 <mark> Converted aufs cache dir into COSS on eiximenis [production]
20:41 <mark> Increased cache dir object size split from 512kb to 1MB [production]
17:45 <domas> readded srv211 to pybal apaches list, either someone removed it by mistake, or didn't document the reason of removal [production]
16:57 <domas> rolling out php5 5.2.4-2ubuntu5.7wm1 and php5-apc 3.1.3p1-1wm1 live on all apaches [production]
15:32 <domas> rolling out limited php 5.2.4-2ubuntu5.7wm1 testing [production]
15:07 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'show draft by default on iawiki' [production]
08:58 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'Set $wgFlaggedRevsAutoconfirm on dewiki' [production]
2009-09-12 §
11:08 <_3^3> srv230-srv250 are running now APC-3.1.3p1-1wm1 [production]
08:53 <_3^3> APC 3.1 is ~3-5% faster than 3.0 under load. Will roll out limited deployment for longer-term stability tests. [production]
08:29 <mark> Rerouted pmtpa->esams traffic via hgtn-leaseweb [production]
2009-09-11 §
19:14 <domas> xcache was both slower and less stable (got corrupted cache within seconds, can we blame that on -O3? :)) [production]
18:56 <domas> (xcache is configured to have ttl of 600 seconds on all php objects.. ;-) [production]
18:55 <domas> installed php5-xcache on srv250 (manual comment in apc.ini though...) [production]
18:39 <fvassard> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling Collection on cswiki per Bug 20436' [production]
06:25 <domas> added api.php to wpstatic acl [production]
2009-09-10 §
21:36 <tomaszf> starting [a-z]* sync of commons images from ms5->belgrade [production]
19:30 <tomaszf> starting [0-9]* sync of commons images from ms5->belgrade [production]
2009-09-09 §
20:16 <domas> unmounted /mnt/scratch on amane, /dev/sdb is gone long time ago. [production]
19:05 <aZaFred_> modified puppetized cron job on the scalers to escape parenthesis in the find command [production]
18:14 <domas> restarted bunch of evil apaches [production]