| 
      
        2014-02-03
      
      §
     | 
  
    
  | 19:11 | 
  <Reedy> | 
  Added org_last_active_date column to ep_orgs on all wikis running EducationProgram - bug 60775 | 
  [production] | 
            
  | 06:56 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'depool db1049 for schema changes' | 
  [production] | 
            
  | 05:44 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'repool db1051, warm up' | 
  [production] | 
            
  | 03:40 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'depool db1051 for schema changes' | 
  [production] | 
            
  | 03:29 | 
  <springle> | 
  synchronized wmf-config/db-pmtpa.php  'depool db1052 for schema changes' | 
  [production] | 
            
  | 02:15 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-03 02:15:14+00:00 | 
  [production] | 
            
  | 02:06 | 
  <LocalisationUpdate> | 
  completed (1.23wmf12) at 2014-02-03 02:06:16+00:00 | 
  [production] | 
            
  | 02:06 | 
  <LocalisationUpdate> | 
  completed (1.23wmf11) at 2014-02-03 02:03:45+00:00 | 
  [production] | 
            
  | 02:06 | 
  <Reedy> | 
  wmflabs bastion /home is giving "Read-only file system" | 
  [production] | 
            
  | 01:27 | 
  <springle> | 
  synchronized wmf-config/db-pmtpa.php  'rotate s4 master db1059 to db1040' | 
  [production] | 
            
  | 01:26 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'rotate s4 master db1059 to db1040' | 
  [production] | 
            
  | 01:08 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'prepare for s4 master rotation db1059 to db1040' | 
  [production] | 
            
  
    | 
      
        2014-02-02
      
      §
     | 
  
    
  | 22:01 | 
  <ori> | 
  synchronized tests/noc-conf/highlightTest.php  'I8352c4cfc: Fix test broken by Idda8cff80 (2/2)' | 
  [production] | 
            
  | 21:59 | 
  <ori> | 
  synchronized docroot/noc/conf/highlight.php  'I8352c4cfc: Fix test broken by Idda8cff80 (1/2)' | 
  [production] | 
            
  | 21:58 | 
  <ori> | 
  updated /a/common to {{Gerrit|I8352c4cfc}}: Fix test broken by Idda8cff80 | 
  [production] | 
            
  | 21:40 | 
  <ori> | 
  synchronized docroot/noc/conf/highlight.php  'Idda8cff80: Replace easter egg by a more explaining message' | 
  [production] | 
            
  | 21:37 | 
  <ori> | 
  updated /a/common to {{Gerrit|Idda8cff80}}: Replace easter egg by a more explaining message | 
  [production] | 
            
  | 20:20 | 
  <awight_> | 
  enabling campaigns: C14_enWW_frm_FR C13_wpdr_enSG_FR C13_wpnd_mlotWW_FR C13_wpnd_zhCN_FR C13_wpnd_mlWW_FR | 
  [production] | 
            
  | 19:03 | 
  <awight_> | 
  disabled campaigns: C14_enWW_frm_FR C13_wpdr_enSG_FR C13_wpnd_mlotWW_FR C13_wpnd_zhCN_FR C13_wpnd_mlWW_FR | 
  [production] | 
            
  | 18:15 | 
  <awight_> | 
  stomp updated from 0fa5283c938b2a551278467371acdf4b491f82de to 361c21946fae4eb91e86ffcf886d7f9b388e1289 | 
  [production] | 
            
  | 15:14 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'repool db1037 and db1004' | 
  [production] | 
            
  | 10:53 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'depool db1037, schema changes' | 
  [production] | 
            
  | 09:52 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'depool db1004, schema changes' | 
  [production] | 
            
  | 05:47 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'repool db1011, warm up' | 
  [production] | 
            
  | 03:51 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-02 03:50:59+00:00 | 
  [production] | 
            
  | 03:11 | 
  <LocalisationUpdate> | 
  completed (1.23wmf12) at 2014-02-02 03:11:33+00:00 | 
  [production] | 
            
  | 02:44 | 
  <LocalisationUpdate> | 
  completed (1.23wmf11) at 2014-02-02 02:44:17+00:00 | 
  [production] | 
            
  
    | 
      
        2014-02-01
      
      §
     | 
  
    
  | 20:20 | 
  <ori> | 
  synchronized php-1.23wmf11/extensions/FeaturedFeeds  'I8a600d37a: Reject FeedItem timestamps set too far in the future' | 
  [production] | 
            
  | 20:19 | 
  <ori> | 
  sync-dir: mw1054: No route to host; mw1073: Connection timed out | 
  [production] | 
            
  | 20:19 | 
  <ori> | 
  synchronized php-1.23wmf12/extensions/FeaturedFeeds  'I8a600d37a: Reject FeedItem timestamps set too far in the future' | 
  [production] | 
            
  | 12:21 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'db1052 increase LB' | 
  [production] | 
            
  | 11:26 | 
  <paravoid> | 
  restarting gdnsd on ns0/1/2 to reload maxmind geoip databases (FOSDEM geolocated esams, among others ;)) | 
  [production] | 
            
  | 02:25 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-01 02:25:34+00:00 | 
  [production] | 
            
  | 02:17 | 
  <LocalisationUpdate> | 
  completed (1.23wmf12) at 2014-02-01 02:17:03+00:00 | 
  [production] | 
            
  | 02:02 | 
  <LocalisationUpdate> | 
  completed (1.23wmf11) at 2014-02-01 02:02:04+00:00 | 
  [production] | 
            
  
    | 
      
        2014-01-31
      
      §
     | 
  
    
  | 23:04 | 
  <awight_> | 
  updated tools to 7c4114f3db978996a460d154dcd5da2bd0ffb48d | 
  [production] | 
            
  | 23:03 | 
  <awight_> | 
  updated crm from 99c1702dee9ff230eabb001cd143e0e553a5e8a0 to 95665543297b4f6016e443bcd2194fa5e0f0c96f | 
  [production] | 
            
  | 21:47 | 
  <ori> | 
  finished scap: scap 1.23wmf12 for wikidata i18n changes (duration: 14m 26s) | 
  [production] | 
            
  | 21:37 | 
  <demon> | 
  synchronized php-1.23wmf12/extensions/LiquidThreads/pages/TalkpageView.php  'Fix for broken search bar' | 
  [production] | 
            
  | 21:36 | 
  <demon> | 
  synchronized php-1.23wmf11/extensions/LiquidThreads/pages/TalkpageView.php  'Fix for broken search bar' | 
  [production] | 
            
  | 21:35 | 
  <ori> | 
  started scap: scap 1.23wmf12 for wikidata i18n changes | 
  [production] | 
            
  | 21:29 | 
  <ori> | 
  synchronized php-1.23wmf12/extensions/Wikidata  'Iab3e51adf, I1364bfc86, I25dbed450: bug-fixes for 60670' | 
  [production] | 
            
  | 20:40 | 
  <mwalker|alt> | 
  updating civicrm from 600e09e35c59c3b8c127e5f3c3cf2df11db4f20e to deploy master for thank you messages | 
  [production] | 
            
  | 20:04 | 
  <ori> | 
  stopping sysv parsoid service on wtp1015 to test I74ba5f649 | 
  [production] | 
            
  | 16:56 | 
  <ottomata> | 
  rebooting analytics1003 | 
  [production] | 
            
  | 16:43 | 
  <ottomata> | 
  rebooting an03 to test kafka ulmits | 
  [production] | 
            
  | 16:22 | 
  <ottomata> | 
  reenabling puppet on cp3019 | 
  [production] | 
            
  | 16:01 | 
  <ottomata> | 
  stopping puppet on cp3019 to experiement with varnishkafka buffer levels | 
  [production] | 
            
  | 14:45 | 
  <ottomata> | 
  analytics1022 back up with higher nofile ulimit, now handling all kafka traffic, analytics1021 wont' boot | 
  [production] | 
            
  | 14:10 | 
  <ottomata> | 
  upping noflie open file limit on analytics1021 and analytics1022, rebooting | 
  [production] |