| 2012-11-14
      
      § | 
    
  | 21:41 | <logmsgbot_> | pgehres synchronized php-1.21wmf4/extensions/ContributionReporting/ | [production] | 
            
  | 21:32 | <notpeter> | repooling srv200-srv213 due to lack of automated imaging :( | [production] | 
            
  | 21:19 | <binasher> | restarted lsearchd on all pmtpa lucene hosts | [production] | 
            
  | 20:44 | <notpeter> | removing srv200-srv213 from apaches pool for upgarde to precise | [production] | 
            
  | 20:42 | <binasher> | fixed private wiki search index builds in pmtpa (standby cluster) and building initial wikivoyagedata indices there | [production] | 
            
  | 20:38 | <Reedy> | Killing /tmp/mw-cache-1.20wmf* from all apaches | [production] | 
            
  | 20:20 | <mutante> | installing some package upgrades on sodium (lists) | [production] | 
            
  | 20:10 | <Reedy> | Running sync-common on mw53 | [production] | 
            
  | 19:56 | <Reedy> | srv269 has a full /tmp. Cleaning up | [production] | 
            
  | 19:55 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else (non wikipedia) to 1.21wmf4 | [production] | 
            
  | 19:52 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf4 | [production] | 
            
  | 19:46 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, readonly, closed and special to 1.21wmf4 | [production] | 
            
  | 19:40 | <paravoid> | depooling ms-fe1 for rewrite.py testing | [production] | 
            
  | 19:03 | <Reedy> | Running sync-common on mw48 | [production] | 
            
  | 18:28 | <notpeter> | temp stopping puppet on brewster | [production] | 
            
  | 18:24 | <mutante> | patching Bugzilla to 4.0.9 | [production] | 
            
  | 17:56 | <logmsgbot_> | aaron synchronized php-1.21wmf3/extensions/TimedMediaHandler  'deployed 275aa9524ea6e1e15d7eef3d81c68dde3ffd5693' | [production] | 
            
  | 17:46 | <Reedy> | oai.oaiaudit cleaned out on S3 | [production] | 
            
  | 16:39 | <RobH> | server wmf5815 named 'otto' after Jan Otto, name suggested by DannyB | [production] | 
            
  | 16:22 | <logmsgbot_> | py synchronized wmf-config/db-secondary.php  'updating comments in db-secondary.php' | [production] | 
            
  | 12:52 | <mark> | Reinstall ms-be3001 | [production] | 
            
  | 04:11 | <Tim> | on zhwiki: removing refreshLinks2 jobs for templates "Side box", "Sister", "Flag/core" | [production] | 
            
  | 03:25 | <Tim> | on zhwiki: removing 970k country data jobs | [production] | 
            
  | 02:50 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf4) at Wed Nov 14 02:50:04 UTC 2012 | [production] | 
            
  | 02:29 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf3) at Wed Nov 14 02:29:53 UTC 2012 | [production] | 
            
  | 00:54 | <logmsgbot_> | aaron synchronized php-1.21wmf4/extensions/TimedMediaHandler  'deployed 2784d7ab1b0e0a56b905ff4b9c4070f63f867d25' | [production] | 
            
  | 00:39 | <logmsgbot_> | aaron synchronized wmf-config/filebackend.php  'Set "swiftTempUrlKey"' | [production] | 
            
  | 00:28 | <AaronSchulz> | Set X-Account-Meta-Temp-Url-Key for swift mw:thumb account | [production] | 
            
  | 00:27 | <logmsgbot_> | aaron synchronized wmf-config/PrivateSettings.php | [production] | 
            
  | 00:21 | <logmsgbot_> | asher synchronized wmf-config/mc.php  'disable general memcached multiwriting: nothing but pecl' | [production] | 
            
  | 00:19 | <logmsgbot_> | asher synchronized wmf-config/CommonSettings.php  'remove apache memchached's from the parsercache multiwriter' | [production] | 
            
  
    | 2012-11-13
      
      § | 
    
  | 22:56 | <logmsgbot_> | kaldari synchronized wmf-config/PrivateSettings.php  'syncing PrivateSettings back to previous version' | [production] | 
            
  | 22:41 | <binasher> | converted puppet.updates_update table to innodb - received the most number of of any puppet db table and was myisam, yikes. this might help with concurrency. | [production] | 
            
  | 22:31 | <LeslieCarr> | restarting gmetad on nickel | [production] | 
            
  | 22:22 | <Reedy> | Publishing schema for lucene/oai on mw.o.... | [production] | 
            
  | 22:07 | <mutante> | mailman lists: renaming wikimedia-th to wikimedia-th-private | [production] | 
            
  | 22:06 | <logmsgbot_> | kaldari synchronized wmf-config/PrivateSettings.php  'syncing PrivateSettings for Flickr API key' | [production] | 
            
  | 18:43 | <RobH> | analytics1007 now pxe boots | [production] | 
            
  | 18:41 | <RobH> | analytics1007.mgmt locked up, had to remove all power to fix, now working | [production] | 
            
  | 18:33 | <RobH> | testing analytics1002, pxe boots no problem | [production] | 
            
  | 17:53 | <paravoid> | setting up media1:thumbs on nas1001-a and semi-sync snapmirror from nas1-a | [production] | 
            
  | 17:24 | <Jeff_Green> | wikimedia.org zone update, new DKIM keys for silverpop | [production] | 
            
  | 16:54 | <paravoid> | snapmirror between nas1-a/nas1001-a for all volumes in-sync again | [production] | 
            
  | 16:50 | <paravoid> | running "cf giveback" on nas1001-b to restore nas1001-a; had to "partner snapmirror off" first. | [production] | 
            
  | 16:46 | <apergos> | had to restart varnish frontend by hand after reboot, also the varnishncsa for emery, locke, multicast | [production] | 
            
  | 16:30 | <apergos> | rebooting cp1042 (XFS: possible memory allocation deadlock in kmem_alloc), lots of stuck /bin/ps form nagios, load average ridiculous | [production] | 
            
  | 16:10 | <paravoid> | creating media1:thumbs volume at nas1-a and setting up /etc/export rules for it | [production] | 
            
  | 16:08 | <cmjohnson1> | provision labsdb3 | [production] | 
            
  | 15:32 | <cmjohnson1> | provisioning labsdb2 | [production] | 
            
  | 15:15 | <hashar> | restarting jenkins on gallium | [production] |