| 
      
        2014-01-23
      
      §
     | 
  
    
  | 04:10 | 
  <reedy> | 
  synchronized docroot/noc/conf/index.php | 
  [production] | 
            
  | 04:07 | 
  <reedy> | 
  synchronized multiversion/ | 
  [production] | 
            
  | 04:05 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 04:04 | 
  <reedy> | 
  updated /a/common to {{Gerrit|I06964b159}}: dirname( __FILE__ ) to __DIR__ | 
  [production] | 
            
  | 02:18 | 
  <reedy> | 
  synchronized php-1.23wmf11/extensions/EducationProgram/ | 
  [production] | 
            
  | 02:14 | 
  <ori> | 
  synchronized wmf-config/InitialiseSettings.php  'I0dbeff07d: : true everywhere' | 
  [production] | 
            
  | 02:13 | 
  <ori> | 
  updated /a/common to {{Gerrit|I0dbeff07d}}: $wgResourceLoaderStorageEnabled: true everywhere | 
  [production] | 
            
  | 02:10 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-01-23 02:09:57+00:00 | 
  [production] | 
            
  | 02:03 | 
  <LocalisationUpdate> | 
  completed (1.23wmf11) at 2014-01-23 02:03:47+00:00 | 
  [production] | 
            
  | 02:02 | 
  <ori> | 
  synchronized wmf-config/InitialiseSettings.php  'I16738b6ef: : true everywhere except enwiki' | 
  [production] | 
            
  | 02:02 | 
  <LocalisationUpdate> | 
  completed (1.23wmf10) at 2014-01-23 02:02:42+00:00 | 
  [production] | 
            
  | 02:02 | 
  <ori> | 
  updated /a/common to {{Gerrit|I16738b6ef}}: $wgResourceLoaderStorageEnabled: true everywhere except enwiki | 
  [production] | 
            
  | 01:08 | 
  <ori> | 
  synchronized wmf-config/InitialiseSettings.php  'I321ce06ae: Enable module storage on it, de, es and nlwikis' | 
  [production] | 
            
  | 01:08 | 
  <ori> | 
  updated /a/common to {{Gerrit|I321ce06ae}}: Enable module storage on it, de, es and nlwikis | 
  [production] | 
            
  | 00:56 | 
  <hasharMeeting> | 
  integration.wikimedia.org and doc.wikimedia.org have been successfully migrated behind the misc varnish | 
  [production] | 
            
  | 00:53 | 
  <ori> | 
  doing staggered graceful restarts on bits app servers for I9e311c98f | 
  [production] | 
            
  | 00:38 | 
  <hasharMeeting> | 
  migrating  (with RobH) integration.wikimedia.org being misc varnish | 
  [production] | 
            
  | 00:24 | 
  <hasharMeeting> | 
  restarted zuul again | 
  [production] | 
            
  | 00:18 | 
  <hasharMeeting> | 
  upgrading Zuul (lightning deploy of doom) | 
  [production] | 
            
  
    | 
      
        2014-01-22
      
      §
     | 
  
    
  | 23:41 | 
  <hasharMeeting> | 
  Going to upgrade Zuul, need a patch to make it send  Cache-Control: no-cache so we can migrate Zuul service behind misc varnish (being done with RobH) | 
  [production] | 
            
  | 22:58 | 
  <ottomata> | 
  restarted icinga (a while ago) and puppet freshness checks are all boogery(?) will check up on this tomorrow | 
  [production] | 
            
  | 22:51 | 
  <RobH> | 
  doc.wikimedia.org updates complete, once dns re-propagates it should be again reachable over https | 
  [production] | 
            
  | 22:43 | 
  <RobH> | 
  doc.wikimedia.org and doc.mediawiki.org relocating behind misc-web-lb.  dns ttl was set to 5 minutes yesterday, so this change should result in less than 5 minutes of unreachability by anyone | 
  [production] | 
            
  | 22:09 | 
  <reedy> | 
  updated /a/common to {{Gerrit|Ie95fdf8e9}}: Adjust linkpurge and renderfile limits | 
  [production] | 
            
  | 21:26 | 
  <matanya> | 
  git.wikimedia.org is back up | 
  [production] | 
            
  | 20:44 | 
  <RobH> | 
  all plugins updated except caching cuz i hate it and i plan to move it behind misc-web-lb | 
  [production] | 
            
  | 20:42 | 
  <RobH> | 
  blog updated | 
  [production] | 
            
  | 19:40 | 
  <matanya> | 
  git.wikimedia.org is down | 
  [production] | 
            
  | 19:32 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 17:40 | 
  <RobH> | 
  virt0 puppet agent re-enabled and resumed normal service | 
  [production] | 
            
  | 17:05 | 
  <RobH> | 
  issues with change, don't re-enable puppet agent on virt0 yet | 
  [production] | 
            
  | 16:59 | 
  <RobH> | 
  rolling certificate changes to virt1000 and virt0.  virt0 puppet agent disabled for moment | 
  [production] | 
            
  | 16:33 | 
  <Nemo_bis> | 
  still no data in Global_JobQueue_length ganglia graph on hume (3 days now) | 
  [production] | 
            
  | 14:34 | 
  <csteipp> | 
  synchronized php-1.23wmf10/includes/media  'bug60339' | 
  [production] | 
            
  | 14:33 | 
  <csteipp> | 
  synchronized php-1.23wmf11/includes/media  'bug60339' | 
  [production] | 
            
  | 11:18 | 
  <apergos> | 
  powercycled kaulen, it had died the swap death | 
  [production] | 
            
  | 10:02 | 
  <apergos> | 
  restarted mw1206 a little while ago, it was inaccessible even via mgmt | 
  [production] | 
            
  | 06:31 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 05:37 | 
  <ori> | 
  restarting hafnium for kernel upgrade | 
  [production] | 
            
  | 03:33 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-01-22 03:33:21+00:00 | 
  [production] | 
            
  | 00:55 | 
  <marktraceur> | 
  deployed config for jsduck jobs for MultimediaViewer | 
  [production] | 
            
  | 00:22 | 
  <demon> | 
  synchronized php-1.23wmf11/extensions/CirrusSearch  'Cirrus to master, with better job queues' | 
  [production] | 
            
  | 00:21 | 
  <demon> | 
  synchronized php-1.23wmf10/extensions/CirrusSearch  'Cirrus to master, with better job queues' | 
  [production] | 
            
  | 00:18 | 
  <demon> | 
  synchronized php-1.23wmf10/extensions/Elastica  'Elastica to master' | 
  [production] | 
            
  | 00:15 | 
  <demon> | 
  synchronized php-1.23wmf10/extensions/CirrusSearch  'Rolling back Cirrus -- fatals' | 
  [production] | 
            
  | 00:11 | 
  <demon> | 
  synchronized php-1.23wmf10/extensions/CirrusSearch  'Cirrus to master, with better job queues' | 
  [production] |