|
2011-08-12
§
|
| 15:08 |
<mark> |
Turning off ethernet hw offloading GRO on all lvs servers with Puppet |
[production] |
| 14:30 |
<mark> |
Turned off all forms of hardware segmentation on lvs4, fixing the slow upload problem |
[production] |
| 14:29 |
<mark> |
Turned tcp segment offloading back on on sq51..86 |
[production] |
| 13:57 |
<mark> |
Manually turned off TCP segmentation offloading on sq51..86 |
[production] |
| 12:59 |
<catrope> |
synchronized wmf-config/StartProfiler.php 'Remove upload profiling, hasn't produced any useful data' |
[production] |
| 12:52 |
<catrope> |
synchronized wmf-config/StartProfiler.php 'Profile uploads on officewiki separately, I wanna try something' |
[production] |
| 11:39 |
<apergos> |
reran ppuppet by hand on spence, sq32 entries in nagios conf files were not recreated, restarted nagios, seems to be running |
[production] |
| 11:11 |
<apergos> |
er... because sq32 is in the decommissioned list but the script to purge resources from nagios is broken right now, which means nagios fails to start |
[production] |
| 11:11 |
<apergos> |
purged sq32 resources and host references from puppet db manually on db9, and from nagios conf files on spence. will run puppet manually on spence shortly |
[production] |
| 10:52 |
<Andrew> |
(screen on hume) |
[production] |
| 10:52 |
<Andrew> |
Running populatePifEditCount.php on all wikis |
[production] |
| 10:45 |
<Andrew> |
Adding pif_edits table to all wikis for personal image filter voter list |
[production] |
| 10:39 |
<mark> |
Enabled cr1-sdtpa:xe-0/0/2; a cross connect has been ordered, expect Nagios to complain |
[production] |
| 10:24 |
<apergos> |
uncommented the monitor_group line in varnish.pp which defines the cache_mobile_eqiad group in puppet (thanks ma rk), will run puppet shortly on spence |
[production] |
| 08:44 |
<apergos> |
revert change to site.pp, try applying to spence |
[production] |
| 08:29 |
<apergos> |
doing repeated manual runs of puppet on spence til we catch up to current config (it is quite out of date) |
[production] |
| 07:21 |
<apergos> |
nagios was failing to start because of unknown host group cache_mobile_eqiad in /etc/nagios/puppet_hosts.cfg; commented out line $nagios_group = "cache_mobile_${site}" in site.pp, waiting for puppet run to complete on spence |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.17) at Fri Aug 12 02:17:38 UTC 2011 |
[production] |
|
2011-08-11
§
|
| 21:24 |
<JeLuF> |
added 'ttf-ubuntu-font-family' to the list of required packages for image scalers in puppet ([[bugzilla:30288|bug 30288]]) |
[production] |
| 21:07 |
<JeLuF> |
virt2 root filesystem has switched to read-only due to a disk failure |
[production] |
| 21:04 |
<LocalisationUpdate> |
completed (1.17) at Thu Aug 11 21:06:09 UTC 2011 |
[production] |
| 20:24 |
<binasher> |
re-pooling mobile2 |
[production] |
| 20:16 |
<LocalisationUpdate> |
failed |
[production] |
| 18:57 |
<binasher> |
depooling mobile2 from lvs for mobile extension opt in proxy conf testing |
[production] |
| 18:19 |
<preilly> |
pushing new mobile frontend changes to production |
[production] |
| 18:19 |
<preilly> |
synchronizing Wikimedia installation... Revision: 94253: |
[production] |
| 18:12 |
<mark> |
Temporarily serving li.wikipedia.org from srv153 (bypassing LVS) as backend on the text squids |
[production] |
| 18:02 |
<mark> |
Test complete, change reverted |
[production] |
| 17:57 |
<mark> |
Temporarily moved squids->apaches LVS traffic from lvs4 to lvs3 for testing |
[production] |
| 16:48 |
<Ryan_Lane> |
upping the nginx upload size too 100m for the https and ipv6 cluster |
[production] |
| 16:06 |
<Reedy> |
SVN and related services should be ok now. High HTTP load causing OOM |
[production] |
| 15:51 |
<Reedy> |
SVN may be unavailable due to issues with Formey |
[production] |
| 15:32 |
<catrope> |
synchronized php/includes/filerepo/LocalFile.php '[[rev:94252|r94252]] by Chad - Try wrapping ss_images update in a transaction' |
[production] |
| 15:17 |
<RoanKattouw> |
All file uploads were returning HTTP 500 errors between 15:07 and 15:16 UTC, my apologies. It's fixed now |
[production] |
| 15:16 |
<catrope> |
synchronized wmf-config/StartProfiler.php 'Fix it for real this time' |
[production] |
| 15:15 |
<catrope> |
synchronized wmf-config/StartProfiler.php 'Unbreak uploads, oops' |
[production] |
| 15:07 |
<catrope> |
synchronized wmf-config/StartProfiler.php 'Make upload profiling 1:1 instead of 1:50' |
[production] |
| 15:01 |
<mark> |
Set maximum TCP window size on nas1-a |
[production] |
| 15:00 |
<RoanKattouw> |
...and that worked. Lesson of the day: it seems you can't use dashes in your profile IDs |
[production] |
| 14:59 |
<catrope> |
synchronized wmf-config/StartProfiler.php 'Remove dashes from profile IDs just because I'm paranoid' |
[production] |
| 14:56 |
<catrope> |
synchronized wmf-config/StartProfiler.php 'Add profiling groups upload-commons and upload-other, based on count($_FILES)' |
[production] |
| 14:10 |
<mark> |
Setup semi-sync snapmirror replication from nas1001-a:images to nas1-a:images, started initial transfer |
[production] |
| 13:46 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 8886 - Install DynamicPageList extension for Vietnamese Wiktionary' |
[production] |
| 13:37 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:18390|bug 18390]] Set [bureaucrat][] = sysop on enwiki' |
[production] |
| 13:34 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:30307|bug 30307]] Allow bureaucrats to remove sysop flag on Russian Wikipedia' |
[production] |
| 12:45 |
<mark> |
Migrated nas1-a and nas1-b to new 64 bit root volumes |
[production] |
| 09:45 |
<catrope> |
synchronized wmf-config/checkers.php 'Remove broken profiling in wfACLBlocks. Not needed because everything in $wgExtensionFunctions is automatically profiled anyway' |
[production] |
| 09:17 |
<mutante> |
sq75 - power cycle, squid clean |
[production] |
| 03:00 |
<LocalisationUpdate> |
completed (1.17) at Thu Aug 11 03:02:09 UTC 2011 |
[production] |
| 02:32 |
<LocalisationUpdate> |
completed (1.17) at Thu Aug 11 02:34:20 UTC 2011 |
[production] |