7401-7450 of 10000 results (29ms)
2010-10-24 §
12:31 <mark> Powercycled knsq4 [production]
12:31 <mark> Restarted varnish on knsq2 [production]
12:31 <mark> Removed puppet lock on knsq2 [production]
2010-10-23 §
20:48 <domas> powercycled srv154, saw mgmt network problems yet again [production]
20:30 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning of CentralNotice for qualitywiki' [production]
18:32 <RobH> modifying the sync-common-all to take out logging while I am futzing with it. [production]
18:27 <root> ran sync-common-all [production]
18:24 <root> ran sync-common-all [production]
18:21 <root> ran sync-common-all [production]
18:19 <root> ran sync-common-all [production]
18:17 <root> ran sync-common-all [production]
18:13 <domas> increased maxclients 10->20 on scalers [production]
18:10 <root> ran sync-common-all [production]
17:37 <Ryan_Lane> updating the dns entry for svn.wikimedia.org [production]
17:37 <Ryan_Lane> rsynd'd /svnroot to formey [production]
17:37 <Ryan_Lane> remounted /svnroot ro on mayflower [production]
17:37 <Ryan_Lane> moving svn from mayflower to formey [production]
17:03 <domas> restarted apaches that failed to start after apt-get upgrade :( [production]
17:01 <root> ran sync-common-all [production]
16:58 <RobH> working on rsync settings and running crap as root for testing, disregard [production]
16:57 <tomaszf> upgrading activemq to 5.4.1 on erzurumi [production]
16:51 <tstarling> synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression' [production]
16:48 <tstarling> synchronized php-1.5/includes/memcached-client.php 'profiling memcached compression' [production]
16:46 <catrope> synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php 'r75268' [production]
16:36 <root> ran sync-common-all [production]
16:33 <domas> upgrading scalers to new imagemagick and whatever is pending [production]
16:20 <RobH> having to run the sync as root to fix permissions with roan, not normal use [production]
16:19 <root> ran sync-common-all [production]
16:10 <root> ran sync-common-all [production]
15:22 <mark> Deployed wikimedia.vcl (varnish) which fixes a memleak in the geoip code [production]
14:27 <catrope> synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 'r75239' [production]
14:09 <Ryan_Lane> updated nova-* on ersch and alsted for stable release - rebooted [production]
11:15 <mark> Unstuck srv220 and srv222 [production]
2010-10-22 §
17:25 <RobH> at this time no issues are apparent and site access appears normal again. [production]
17:25 <RobH> Reported issues with accessing cluster hitting esams, possible network issues, further investigation shows no server errors. If network links continue to cause issues, we may redirect traffic stateside [production]
02:36 <apergos> shot some hung converts on the image scalers, that should hold us for awhile [production]
2010-10-21 §
21:10 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'Whitelisting new landing pages' [production]
20:37 <Ryan_Lane> adding account for jdavis on mchenry and sanger [production]
18:31 <mark> Powercycled sq67 and sq68 [production]
18:10 <mark> Unstuck srv219, 223, 224 [production]
18:05 <AaronSchulz> ran runBatchedQuery.php to finish enwiki rename 'Voice of All' => 'Aaron Schulz' on archive/revision tables [production]
17:58 <Ryan_Lane|food> enabled eth0 interface on test-payments, and changed the IP to the new IP [production]
17:56 <RobH> added dns entry for test-payments on prototype server [production]
16:47 <catrope> synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r75158' [production]
15:26 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' [production]
15:14 <RoanKattouw> Commented out srv154 in /etc/dsh/groups/mediawiki-installation because it's down and blocking sync-file [production]
15:13 <catrope> synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r75156' [production]
01:02 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'Readding payments sidebar links after outage' [production]
2010-10-20 §
22:03 <mark> Removed squid.conf log line sending packets to yaseo [production]
21:54 <RobH> tired of misdated cron emails from dataset1, fixed the date again, it will go off date on its own eventually [production]