9551-9600 of 10000 results (26ms)
2010-10-25 §
16:09 <apergos> after restart of mysqld on db9, it now accepts requests from singer, blogs appear to be back [production]
15:41 <apergos> powercycled singer, it was unresponsive to ssh, huge cpu spikes [production]
2010-10-24 §
20:44 <mark> Shutting down mayflower for decommissioning [production]
20:40 <mark> Changed root password on lily, iris, ragweed, mayflower and sage [production]
20:37 <mark> Changed root password on browne and srv2 [production]
19:31 <RobH> installed python-openssl on spence for testing [production]
19:11 <tomaszf> removing downtime with init script on loudon [production]
18:21 <midom> synchronized php-1.5/wmf-config/db.php [production]
18:20 <midom> synchronized php-1.5/wmf-config/db.php [production]
17:44 <Ryan_Lane> reinstalling ersch for openstack testing [production]
17:29 <mark> Changed root password hash in puppet; being deployed to all systems in the next 30-60 minutes [production]
17:21 <midom> synchronized php-1.5/wmf-config/db.php [production]
16:56 <Ryan_Lane> reinstalling alsted, to try stable openstack from scratch [production]
16:43 <Ryan_Lane> rebooting ersch and alsted [production]
14:26 <midom> synchronized php-1.5/wmf-config/db.php [production]
14:26 <midom> synchronized php-1.5/wmf-config/db.php [production]
12:34 <mark> Restarted varnish on knsq5 [production]
12:33 <mark> Restarted varnish on knsq1 [production]
12:31 <mark> Powercycled knsq4 [production]
12:31 <mark> Restarted varnish on knsq2 [production]
12:31 <mark> Removed puppet lock on knsq2 [production]
2010-10-23 §
20:48 <domas> powercycled srv154, saw mgmt network problems yet again [production]
20:30 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning of CentralNotice for qualitywiki' [production]
18:32 <RobH> modifying the sync-common-all to take out logging while I am futzing with it. [production]
18:27 <root> ran sync-common-all [production]
18:24 <root> ran sync-common-all [production]
18:21 <root> ran sync-common-all [production]
18:19 <root> ran sync-common-all [production]
18:17 <root> ran sync-common-all [production]
18:13 <domas> increased maxclients 10->20 on scalers [production]
18:10 <root> ran sync-common-all [production]
17:37 <Ryan_Lane> updating the dns entry for svn.wikimedia.org [production]
17:37 <Ryan_Lane> rsynd'd /svnroot to formey [production]
17:37 <Ryan_Lane> remounted /svnroot ro on mayflower [production]
17:37 <Ryan_Lane> moving svn from mayflower to formey [production]
17:03 <domas> restarted apaches that failed to start after apt-get upgrade :( [production]
17:01 <root> ran sync-common-all [production]
16:58 <RobH> working on rsync settings and running crap as root for testing, disregard [production]
16:57 <tomaszf> upgrading activemq to 5.4.1 on erzurumi [production]
16:51 <tstarling> synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression' [production]
16:48 <tstarling> synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression' [production]
16:46 <catrope> synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php 'r75268' [production]
16:36 <root> ran sync-common-all [production]
16:33 <domas> upgrading scalers to new imagemagick and whatever is pending [production]
16:20 <RobH> having to run the sync as root to fix permissions with roan, not normal use [production]
16:19 <root> ran sync-common-all [production]
16:10 <root> ran sync-common-all [production]
15:22 <mark> Deployed wikimedia.vcl (varnish) which fixes a memleak in the geoip code [production]
14:27 <catrope> synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 'r75239' [production]
14:09 <Ryan_Lane> updated nova-* on ersch and alsted for stable release - rebooted [production]