|
2011-08-05
§
|
| 10:06 |
<mark> |
Fixed memcached check on spence |
[production] |
| 09:47 |
<mark> |
srv301 didn't have the right (API) service IPs bound; manually ran dpkg-reconfigure wikimedia-lvs-realserver to fix this |
[production] |
| 04:10 |
<maplebed> |
wikimedia-task-dns-auth package upgraded on ns0 to put zone files in revision control. |
[production] |
| 02:38 |
<aaron> |
synchronized php/maintenance/addwiki.php 'deployed [[rev:93593|r93593]]' |
[production] |
| 02:31 |
<aaron> |
synchronized wmf-config/CommonSettings.php 'renamed versionNumber variables to have wmf prefix' |
[production] |
| 02:25 |
<LocalisationUpdate> |
completed at Fri Aug 5 02:27:52 UTC 2011 |
[production] |
| 00:59 |
<notpeter> |
knsq30 is seems to have some kind of hardware failure. stopping squid and squid frontend proc |
[production] |
| 00:48 |
<AaronSchulz> |
/home/wikipedia/bin version with a working repo again |
[production] |
|
2011-08-04
§
|
| 23:57 |
<binasher> |
deploying mobile squid acls with nokia and netfront enabled for a brief test to see if todays mobile gateway changes will allow it to support the load |
[production] |
| 23:10 |
<binasher> |
squid mobile redirect deployment completed |
[production] |
| 22:38 |
<binasher> |
mobile redirect deploy complete to all pmtpa squids. mobile cluster looks good, starting amsterdam |
[production] |
| 22:17 |
<binasher> |
starting an extra slow rollout of mobile redirect acls to frontend squids, minus nokia and netfront |
[production] |
| 21:43 |
<binasher> |
varnish 3.0 is now on all RoR mobile gw servers - only caching static non-article content |
[production] |
| 21:33 |
<binasher> |
mobile2 is back / varnish for static content / apache relax |
[production] |
| 19:52 |
<binasher> |
pulling mobile2 from lvs for testing |
[production] |
| 17:17 |
<reedy> |
synchronized multiversion/MWMultiVersion.php '[[rev:93902|r93902]]' |
[production] |
| 17:17 |
<reedy> |
synchronized multiversion/MWMultiVersion.php '[[rev:93901|r93901]]' |
[production] |
| 17:12 |
<reedy> |
synchronized multiversion/MWMultiVersion.php '[[rev:93901|r93901]]' |
[production] |
| 17:11 |
<AaronSchulz> |
Fixed thumbnails on testwiki ([[bugzilla:30210|bug 30210]]) |
[production] |
| 17:10 |
<aaron> |
synchronized live-1.5/MWVersion.php |
[production] |
| 15:40 |
<mark> |
Mounted nas1001-a:/vol/images on ms1002:/vol/images for testing, started rsync copy from ms1002:/export/upload to /vol/images |
[production] |
| 15:20 |
<mark> |
Created 12-disk aggregate test0 on nas1001-a |
[production] |
| 15:01 |
<RoanKattouw> |
Staging some HTTPS support / prot rel URL code on testwiki, see today's 1.17wmf1 merges. DO NOT DEPLOY SITEWIDE until tomorrow |
[production] |
| 13:19 |
<RobH> |
updated wikibugs on mchenry about 10 minutes ago and forgot to admin log |
[production] |
| 13:18 |
<mark> |
Created new 64 bit aggregate and root volume on nas1001-a, copied over root data, destroyed old root volume, moving new aggregate to first disks |
[production] |
| 13:00 |
<RobH> |
disabled line numbers in vimrc.local on puppet because i hate line numbers (it already says the line number yer on in the corner) |
[production] |
| 12:21 |
<mark> |
Started disk replace operation of 0a.10.0 by 0a.10.3 and 0a.10.1 by 0a.10.4 on nas1-b |
[production] |
| 12:12 |
<mark> |
Started disk replace operation of 0b.00.0 by 0a.01.1, and 0b.00.1 by 0a.01.2 on nas1-a |
[production] |
| 12:08 |
<mutante> |
ragweed - package updates, restart required |
[production] |
| 11:05 |
<mutante> |
amssq46 - power cycle, clean squid, dist-upgrade/kernel, reboot |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed at Thu Aug 4 02:20:48 UTC 2011 |
[production] |
|
2011-08-03
§
|
| 21:22 |
<binasher> |
previous log line should have read $ppsize + 50 |
[production] |
| 21:21 |
<binasher> |
dropping apache maxclients on mobile servers to ($ppsize - 50) |
[production] |
| 18:24 |
<aaron> |
synchronized multiversion/MWMultiVersion.php 'deployed [[rev:93845|r93845]]' |
[production] |
| 18:23 |
<aaron> |
synchronized multiversion/MWScript.php 'deployed [[rev:93845|r93845]]' |
[production] |
| 16:19 |
<tfinc> |
synchronized wmf-config/InitialiseSettings.php 'testing collectinos on hewiki' |
[production] |
| 16:13 |
<mutante> |
installed Net::Stomp Perl module on spence via cpan shell which fixed RT #703 |
[production] |
| 14:32 |
<RobH> |
updating dns for magnesium and zinc eqiad misc servers to public IPs RT#1245 |
[production] |
| 13:24 |
<tstarling> |
synchronized php-1.17/extensions/CodeReview/ui/WordCloud.php |
[production] |
| 13:24 |
<tstarling> |
synchronized php-1.17/extensions/CodeReview/CodeReview.php |
[production] |
| 12:21 |
<mark> |
Rebooting both nas1001 controllers with corrected network config |
[production] |
| 10:02 |
<root> |
synchronized wmf-config/mc.php 'swapped out all down servers returning 0 with ones returning 100, cleaned up spares and down list to be accurate' |
[production] |
| 09:44 |
<RobH> |
swapping out down memcached servers iwth tested up ones |
[production] |
| 09:44 |
<RobH> |
incorrect, turns out there are some down, they reply but do not return full weight |
[production] |
| 09:44 |
<robh> |
synchronized wmf-config/mc.php 'replacing down servers' |
[production] |
| 09:40 |
<RobH> |
nagios shows spence memcached error, yet manual testing shows all memcached are pooled fine. |
[production] |
| 09:01 |
<Ryan_Lane> |
installing ssl1001-4 |
[production] |
| 08:09 |
<apergos> |
live hack to /usr/share/etherpad/etherpad/bin/run-local.sh on hooper to force maxthreads to 512 for etherpad. restarted etherpad, this may or may not help connection issues (maybe we'll just oom instead?) |
[production] |
| 07:40 |
<domas> |
stupid bot and its quotes, http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ is the url |
[production] |
| 07:39 |
<midom> |
synchronized php/includes/db/DatabaseMysql.php 'livehacking out sql_mode and set charset code - http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ' |
[production] |