| 2014-03-05
      
      § | 
    
  | 17:20 | <demon> | synchronized wmf-config/InitialiseSettings.php  'Cirrus for all the wikiquotes' | [production] | 
            
  | 16:31 | <cmjohnson1> | carbon going down | [production] | 
            
  | 16:16 | <akosiaris> | upgraded php5 packages, php5-wmerrors and libmemcached packages on beta in preparation for full cluster upgrade. This will make puppet unhappy. | [production] | 
            
  | 16:04 | <cmjohnson1> | shutting down mw1165 swapping DIMM | [production] | 
            
  | 14:40 | <ottomata> | starting controlled shutdown (and restart) of kafka broker analytics1022 to bring in new replica.lag settings | [production] | 
            
  | 14:31 | <ottomata> | starting controlled shutdown of kafka broker an21 to reload new replica.lag settings | [production] | 
            
  | 11:12 | <hashar> | jenkins back | [production] | 
            
  | 11:11 | <ori> | updated /a/common to {{Gerrit|I5277d2451}}: Update Schema:Echo revision to 7731316 | [production] | 
            
  | 11:08 | <hashar> | restarting Jenkins which has starving threads | [production] | 
            
  | 10:03 | <hashar> | Jenkins: gallium overloaded with a few java threads taking 100% CPU starving the box :( | [production] | 
            
  | 05:21 | <Ryan_Lane> | finished switch from the perl git-deploy to trigger | [production] | 
            
  | 04:17 | <Ryan_Lane> | switching out git-deploy perl frontend for trebuchet trigger | [production] | 
            
  | 03:41 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-03-05 03:41:14+00:00 | [production] | 
            
  | 03:37 | <springle> | messing with innodb compression on db1007 | [production] | 
            
  | 02:54 | <LocalisationUpdate> | completed (1.23wmf16) at 2014-03-05 02:54:00+00:00 | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.23wmf15) at 2014-03-05 02:27:46+00:00 | [production] | 
            
  | 00:24 | <ori> | synchronized php-1.23wmf15/extensions/CentralNotice  'Update CentralNotice to tip of wmf_deploy for I7d8259fc4' | [production] | 
            
  | 00:17 | <ori> | synchronized php-1.23wmf16/extensions/CentralNotice  'Update CentralNotice to tip of wmf_deploy for I7d8259fc4' | [production] | 
            
  
    | 2014-03-04
      
      § | 
    
  | 23:09 | <mutante|away> | DNS update - removing 'zhen' | [production] | 
            
  | 23:07 | <bsitu> | synchronized wmf-config/CommonSettings.php  'Update Flow cache key to 3.0' | [production] | 
            
  | 23:06 | <bsitu> | updated /a/common to {{Gerrit|I566e00bcc}}: Update flow cache version number | [production] | 
            
  | 22:58 | <ebernhardson> | synchronized php-1.23wmf15/extensions/Flow/ | [production] | 
            
  | 22:55 | <ebernhardson> | synchronized php-1.23wmf16/extensions/Flow/ | [production] | 
            
  | 22:31 | <ori> | GeoIP cookie enabled on text frontend varnish on cp1066. Appears to work well. If it causes issues, roll back by reverting I7e5ca8e54 and running 'service varnish-frontend restart'. | [production] | 
            
  | 22:14 | <ori> | synchronized wmf-config/CommonSettings-labs.php  'Syncing lab config change I5ca0adf39 to prod for cluster consistency' | [production] | 
            
  | 22:12 | <ori> | updated /a/common to {{Gerrit|I5ca0adf39}}: Labs: set $wgCentralGeoScriptURL to false for GeoIP cookie testing | [production] | 
            
  | 20:45 | <hashar> | operations/apache-config.git now has a "betacluster" branch to host the beta cluster apache configuration files {{bug|56395}} | [production] | 
            
  | 20:39 | <catrope> | synchronized wmf-config/CommonSettings.php  'Actually fix VE on officewiki this time' | [production] | 
            
  | 20:38 | <catrope> | updated /a/common to {{Gerrit|I7ff564e4a}}: Really actually fix VE on private wikis | [production] | 
            
  | 19:22 | <bd808> | rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.23wmf16 | [production] | 
            
  | 19:09 | <ottomata> | depooled elastic1007 and elastic1013-1016 per request from manybubbles | [production] | 
            
  | 19:02 | <bd808> | updated /a/common to {{Gerrit|I62b32888b}}: Create an autopatrolled group on itwikiquote | [production] | 
            
  | 18:47 | <ottomata> | s/analytics/elastic in previous log message | [production] | 
            
  | 18:46 | <ottomata> | bringing analytics1007 and analytics1013-1016 into elasticsearch cluster | [production] | 
            
  | 17:21 | <manybubbles> | rebuilding the search index on commons after failing yesterday now that I've deployed a fix to the timeout issue | [production] | 
            
  | 17:18 | <manybubbles> | synchronized php-1.23wmf16/extensions/CirrusSearch/  '3 small fixes including search timeouts' | [production] | 
            
  | 16:13 | <hashar> | gallium : killed leftover jenkins instance {{bug|51817}} "Jenkins init script is crap" | [production] | 
            
  | 16:07 | <hashar> | gallium has been sent to swap | [production] | 
            
  | 15:41 | <akosiaris> | upgraded php5 packages, php5-wmerrors package and libmemcached11 on mw1017. This will make puppet and the corresponding icinga check unhappy. | [production] | 
            
  | 15:05 | <hashar> | reenabling puppet on gallium | [production] | 
            
  | 14:53 | <andrewbogott> | switched wikitech to allow eqiad access, turned off pmtpa instance creation | [production] | 
            
  | 14:53 | <hashar> | stopping puppet on gallium to play with apache configuration | [production] | 
            
  | 13:52 | <hoo> | synchronized wmf-config/InitialiseSettings.php  '{{Gerrit|I62b3288}}' | [production] | 
            
  | 11:19 | <hashar> | Restarting Jenkins, it is stalled .... | [production] | 
            
  | 11:12 | <hashar> | Jenkins web service unavailable, investigating.  Builds should not be affected though since they dont use the web service (but gearman) | [production] | 
            
  | 03:30 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-03-04 03:29:53+00:00 | [production] | 
            
  | 02:46 | <hoo> | synchronized multiversion/activeMWVersions.php  'Comment only change {{Gerrit|I5e68518}}' | [production] | 
            
  | 02:43 | <hoo> | synchronized multiversion/activeMWVersions.php  'Comment only change {{Gerrit' | [production] | 
            
  | 02:43 | <LocalisationUpdate> | completed (1.23wmf16) at 2014-03-04 02:43:17+00:00 | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.23wmf15) at 2014-03-04 02:28:16+00:00 | [production] |