7551-7600 of 10000 results (31ms)
2011-08-14 §
17:49 <JeLuF> rebooted srv183 [production]
17:30 <JeLuF> stopped apache and jobrunner on srv183, logfiles indicate memory problems [production]
17:18 <jeluf> synchronized wmf-config/mc.php 'replace srv183 by srv229' [production]
13:22 <catrope> synchronized wmf-config/InitialiseSettings.php 'Make $wgServer protocol-relative on wikimania2005wiki, for consistency with the other HTTPS experiment wikis' [production]
10:12 <JeLuF> rebooting sq40 using echo b > /proc/sysrq-trigger, syslog shows lots of SCSI errors [production]
09:10 <apergos> stopped and restarted lsearchd on search7. had to shoot the old one, it wouldn't die [production]
02:19 <LocalisationUpdate> completed (1.17) at Sun Aug 14 02:21:23 UTC 2011 [production]
2011-08-13 §
21:23 <RoanKattouw> sq40 keeps flapping in Nagios all the time [production]
10:39 <RoanKattouw> ...on enwiki [production]
10:39 <RoanKattouw> Running a hacked-up version of extensions/ArticleFeedback/populateAFRevisions.php to fix [[bugzilla:30227|bug 30227]] (data corruption in ArticleFeedback tables), using a list of affected revisions produced on the toolserver [production]
02:16 <LocalisationUpdate> completed (1.17) at Sat Aug 13 02:18:42 UTC 2011 [production]
00:22 <notpeter> upgrading squid and squid-frontend on knsq12 and knsq15 [production]
00:19 <notpeter> upgrading squid and squid-frontend on knsq10 and knsq11 [production]
2011-08-12 §
23:15 <preilly> pushing fix for mobile WML view and disable caching on cookie set page [production]
23:14 <preilly> synchronizing Wikimedia installation... Revision: 94253: [production]
22:40 <notpeter> upgrading squid and squid-frontend on knsq8 and knsq9 [production]
21:55 <notpeter> upgrading squid and squid-frontend on amssq59 and amssq62 [production]
21:46 <preilly> updating the mobile frontend extension on production [production]
21:46 <preilly> synchronizing Wikimedia installation... Revision: 94253: [production]
21:31 <maplebed> localization update permissions changes complete and tested. [production]
21:29 <LocalisationUpdate> completed (1.17) at Fri Aug 12 21:31:45 UTC 2011 [production]
21:26 <AaronSchulz> Running LU to confirm permission fix [production]
21:25 <notpeter> upgrading squid and squid-frontend on amssq56 and amssq58 [production]
21:17 <LocalisationUpdate> completed (1.17) at Fri Aug 12 21:19:39 UTC 2011 [production]
21:10 <maplebed> changing ownership of /usr/local/apache/common-local/php-1.17/cache/l10n from nagios to mwdeploy on all affected srv hosts [production]
21:03 <binasher> deploying new squid frontend.conf - bypass mobile redirector in case of trial opt-in/out pages [production]
20:53 <notpeter> upgrading squid and squid-frontend on amssq51 and amssq52 [production]
20:52 <LocalisationUpdate> completed (1.17) at Fri Aug 12 20:54:15 UTC 2011 [production]
18:44 <maplebed> made ariley an account on rt to submit tickets for email list maintenance [production]
18:24 <JeLuF> DNS: added bugs.mediawiki.org as alias to text.wikimedia.org [production]
17:25 <jeluf> synchronized wmf-config/InitialiseSettings.php '30268 - Point eowp logo to Wiki.png' [production]
16:25 <mark> synchronized wmf-config/CommonSettings.php 'Raise multicast ttl from 2 to 8' [production]
16:10 <catrope> synchronized wmf-config/CommonSettings.php 'Set $wgVaryXFPForAPI = true for HTTPS experiment wikis. This splits the API Squid cache between HTTP and HTTPS, fixing cache pollution issues' [production]
15:08 <mark> Turning off ethernet hw offloading GRO on all lvs servers with Puppet [production]
14:30 <mark> Turned off all forms of hardware segmentation on lvs4, fixing the slow upload problem [production]
14:29 <mark> Turned tcp segment offloading back on on sq51..86 [production]
13:57 <mark> Manually turned off TCP segmentation offloading on sq51..86 [production]
12:59 <catrope> synchronized wmf-config/StartProfiler.php 'Remove upload profiling, hasn't produced any useful data' [production]
12:52 <catrope> synchronized wmf-config/StartProfiler.php 'Profile uploads on officewiki separately, I wanna try something' [production]
11:39 <apergos> reran ppuppet by hand on spence, sq32 entries in nagios conf files were not recreated, restarted nagios, seems to be running [production]
11:11 <apergos> er... because sq32 is in the decommissioned list but the script to purge resources from nagios is broken right now, which means nagios fails to start [production]
11:11 <apergos> purged sq32 resources and host references from puppet db manually on db9, and from nagios conf files on spence. will run puppet manually on spence shortly [production]
10:52 <Andrew> (screen on hume) [production]
10:52 <Andrew> Running populatePifEditCount.php on all wikis [production]
10:45 <Andrew> Adding pif_edits table to all wikis for personal image filter voter list [production]
10:39 <mark> Enabled cr1-sdtpa:xe-0/0/2; a cross connect has been ordered, expect Nagios to complain [production]
10:24 <apergos> uncommented the monitor_group line in varnish.pp which defines the cache_mobile_eqiad group in puppet (thanks ma rk), will run puppet shortly on spence [production]
08:44 <apergos> revert change to site.pp, try applying to spence [production]
08:29 <apergos> doing repeated manual runs of puppet on spence til we catch up to current config (it is quite out of date) [production]
07:21 <apergos> nagios was failing to start because of unknown host group cache_mobile_eqiad in /etc/nagios/puppet_hosts.cfg; commented out line $nagios_group = "cache_mobile_${site}" in site.pp, waiting for puppet run to complete on spence [production]