| 2011-10-25
      
      § | 
    
  | 23:46 | <preilly> | weekly mobile frontend update | [production] | 
            
  | 23:46 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js  'weekly update to mobile frontend' | [production] | 
            
  | 23:45 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/javascripts/application.js  'weekly update to mobile frontend' | [production] | 
            
  | 23:45 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css  'weekly update to mobile frontend' | [production] | 
            
  | 23:45 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php  'weekly update to mobile frontend' | [production] | 
            
  | 23:44 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php  'weekly update to mobile frontend' | [production] | 
            
  | 23:44 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php  'weekly update to mobile frontend' | [production] | 
            
  | 23:09 | <reedy> | synchronized php-1.18/extensions/CodeReview/modules/ext.codereview.js  '[[rev:100760|r100760]]' | [production] | 
            
  | 23:08 | <reedy> | synchronized php-1.18/extensions/Collection/js/collection.js  '[[rev:100760|r100760]]' | [production] | 
            
  | 23:08 | <reedy> | synchronized php-1.18/extensions/AbuseFilter/AbuseFilter.php  '[[rev:100760|r100760]]' | [production] | 
            
  | 22:54 | <reedy> | synchronizing Wikimedia installation... : [[rev:100759|r100759]] | [production] | 
            
  | 22:06 | <Tim> | rebooting hume, it went OOM and stopped responding | [production] | 
            
  | 21:03 | <awjr> | synced civicrm installs on grosley and aluminium to [[rev:671|r671]] of the wikimedia repo | [production] | 
            
  | 20:13 | <LeslieCarr> | updating dns | [production] | 
            
  | 18:37 | <LeslieCarr> | puppetizing mw1001-1160 - please ignore all alerts about them for now | [production] | 
            
  | 17:52 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Reenable ClickTracking UDP logger, now pointing to emery' | [production] | 
            
  | 16:52 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Disable ClickTracking UDP logging to offload locke. Collector had already been killed' | [production] | 
            
  | 16:49 | <RoanKattouw> | Killed AFT log collector on locke. Process was a direct child of init (PID 1), command line was /usr/bin/udp2log --config-file=/etc/udp2log/aft -p 842 | [production] | 
            
  | 13:11 | <catrope> | synchronized php-1.18/cache/interwiki-pr.cdb  '[[bugzilla:31428|bug 31428]]' | [production] | 
            
  | 12:18 | <mark> | All eBGP peerings of br1-knams are off now, no more packet loss | [production] | 
            
  | 12:10 | <mark> | Reenabled some eBGP peerings of br1-knams as cr1-esams transit couldn't handle it alone | [production] | 
            
  | 12:00 | <mark> | Shutting down all BGP peerings of br1-knams | [production] | 
            
  | 11:15 | <mutante> | mw64 - zombie apt-get process,updated APT and libpam,syslog: "nrpe invoked oom-killer","Out of memory: kill process ..(apache2)" etc.,check "free swap" in Ganglia | [production] | 
            
  | 10:39 | <mutante> | mw64 - Apache HTTP flapping OK-CRIT-OK.. - high load avg.,swapping, swap Size/Used 975864 | [production] | 
            
  | 09:08 | <mutante> | erzurumi - installed libpam security updates | [production] | 
            
  | 05:59 | <Tim> | on hume: running refreshImageMetadata.php on all wikis | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed (1.18) at Tue Oct 25 02:20:43 UTC 2011 | [production] | 
            
  
    | 2011-10-24
      
      § | 
    
  | 22:36 | <aaron> | synchronized php-1.18/includes/User.php | [production] | 
            
  | 22:31 | <aaron> | synchronized php-1.18/includes  'deployed [[rev:100657|r100657]]' | [production] | 
            
  | 22:26 | <aaron> | synchronized php-1.18/includes  'deployed [[rev:100655|r100655]]' | [production] | 
            
  | 21:57 | <LeslieCarr> | reformatting all of the wm****.eqiad machines | [production] | 
            
  | 21:54 | <Ryan_Lane> | turning off ci2 | [production] | 
            
  | 19:41 | <cmjohnson1> | resetting storage1 to access bios and check setup | [production] | 
            
  | 19:20 | <aaron> | synchronized wmf-config/flaggedrevs.php  'pre-emptive wg(Add/Remove)Groups settings to account for [[rev:100636|r100636]]; should have no visible effect' | [production] | 
            
  | 19:06 | <reedy> | synchronized php-1.18/extensions/FlaggedRevs/FlaggedRevs.php  '[[rev:100635|r100635]]' | [production] | 
            
  | 18:59 | <mutante> | sodium - initial mailman setup, created site list, will now notify ops@ via mail, storing pass in private repo ./files/misc | [production] | 
            
  | 18:36 | <LeslieCarr> | restarting dead nagios on spence | [production] | 
            
  | 17:40 | <nikerabbit> | synchronizing Wikimedia installation... : i18ndeploy [[rev:100623|r100623]] | [production] | 
            
  | 16:05 | <mark> | restarted pdns on nescio | [production] | 
            
  | 14:59 | <notpeter> | pdns_recursor hung on dobson. restarting | [production] | 
            
  | 14:30 | <Jeff_Green> | installed nagios/nsca on spence | [production] | 
            
  | 14:27 | <mutante> | snapshot4 - dist-upgrade/kernel, reboot | [production] | 
            
  | 14:19 | <mark> | Manually fixed package situation on formey | [production] | 
            
  | 14:10 | <mutante> | ekrem - gzipped access.log.1,  can move/delete access.logs older than X months? | [production] | 
            
  | 14:08 | <mutante> | ekrem at 99% disk space ..again | [production] | 
            
  | 13:14 | <mutante> | something fails btw with etherpad using openoffice ([soffice] <defunct>), (was before and after reboot) | [production] | 
            
  | 13:10 | <mutante> | hooper - fsck was forced and OK, back up, blog and etherpad reachable again | [production] | 
            
  | 13:06 | <mutante> | hooper - dist-upgrade/kernel, reboot | [production] | 
            
  | 13:03 | <mutante> | hooper - temp. shutting down blog and etherpad for kernel update/reboot | [production] | 
            
  | 11:34 | <mutante> | added TERM=dumb to sudo line in /home/w/bin/scap to prevent "unknown terminal type" errors (RT 1767) | [production] |