9351-9400 of 10000 results (39ms)
2010-10-24 §
19:11 <tomaszf> removing downtime with init script on loudon [production]
18:21 <midom> synchronized php-1.5/wmf-config/db.php [production]
18:20 <midom> synchronized php-1.5/wmf-config/db.php [production]
17:44 <Ryan_Lane> reinstalling ersch for openstack testing [production]
17:29 <mark> Changed root password hash in puppet; being deployed to all systems in the next 30-60 minutes [production]
17:21 <midom> synchronized php-1.5/wmf-config/db.php [production]
16:56 <Ryan_Lane> reinstalling alsted, to try stable openstack from scratch [production]
16:43 <Ryan_Lane> rebooting ersch and alsted [production]
14:26 <midom> synchronized php-1.5/wmf-config/db.php [production]
14:26 <midom> synchronized php-1.5/wmf-config/db.php [production]
12:34 <mark> Restarted varnish on knsq5 [production]
12:33 <mark> Restarted varnish on knsq1 [production]
12:31 <mark> Powercycled knsq4 [production]
12:31 <mark> Restarted varnish on knsq2 [production]
12:31 <mark> Removed puppet lock on knsq2 [production]
2010-10-23 §
20:48 <domas> powercycled srv154, saw mgmt network problems yet again [production]
20:30 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning of CentralNotice for qualitywiki' [production]
18:32 <RobH> modifying the sync-common-all to take out logging while I am futzing with it. [production]
18:27 <root> ran sync-common-all [production]
18:24 <root> ran sync-common-all [production]
18:21 <root> ran sync-common-all [production]
18:19 <root> ran sync-common-all [production]
18:17 <root> ran sync-common-all [production]
18:13 <domas> increased maxclients 10->20 on scalers [production]
18:10 <root> ran sync-common-all [production]
17:37 <Ryan_Lane> updating the dns entry for svn.wikimedia.org [production]
17:37 <Ryan_Lane> rsynd'd /svnroot to formey [production]
17:37 <Ryan_Lane> remounted /svnroot ro on mayflower [production]
17:37 <Ryan_Lane> moving svn from mayflower to formey [production]
17:03 <domas> restarted apaches that failed to start after apt-get upgrade :( [production]
17:01 <root> ran sync-common-all [production]
16:58 <RobH> working on rsync settings and running crap as root for testing, disregard [production]
16:57 <tomaszf> upgrading activemq to 5.4.1 on erzurumi [production]
16:51 <tstarling> synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression' [production]
16:48 <tstarling> synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression' [production]
16:46 <catrope> synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php 'r75268' [production]
16:36 <root> ran sync-common-all [production]
16:33 <domas> upgrading scalers to new imagemagick and whatever is pending [production]
16:20 <RobH> having to run the sync as root to fix permissions with roan, not normal use [production]
16:19 <root> ran sync-common-all [production]
16:10 <root> ran sync-common-all [production]
15:22 <mark> Deployed wikimedia.vcl (varnish) which fixes a memleak in the geoip code [production]
14:27 <catrope> synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 'r75239' [production]
14:09 <Ryan_Lane> updated nova-* on ersch and alsted for stable release - rebooted [production]
11:15 <mark> Unstuck srv220 and srv222 [production]
2010-10-22 §
17:25 <RobH> at this time no issues are apparent and site access appears normal again. [production]
17:25 <RobH> Reported issues with accessing cluster hitting esams, possible network issues, further investigation shows no server errors. If network links continue to cause issues, we may redirect traffic stateside [production]
02:36 <apergos> shot some hung converts on the image scalers, that should hold us for awhile [production]
2010-10-21 §
21:10 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'Whitelisting new landing pages' [production]
20:37 <Ryan_Lane> adding account for jdavis on mchenry and sanger [production]