| 2014-02-14
      
      § | 
    
  | 21:25 | <awight_> | crm updated from bd973432c957b6253f82b7b0590251402e75a0de to 20684a7e5dd4270e615ca52e6a17b25e880e47c4 | [production] | 
            
  | 20:46 | <Reedy> | ULSFO down, traffic to Asia etc affected. Being worked on | [production] | 
            
  | 20:43 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Revert Add local interwiki for metawiki' | [production] | 
            
  | 20:10 | <mwalker> | updated payments wiki from from bad46aad4b28435d10570aac8173a7f8dca8751d to 9b936320b797bd01c4e61b1cd7c2e15b0820a24b for version stamp and other 1.22.3 changes | [production] | 
            
  | 20:01 | <ori> | synchronized php-1.23wmf13/extensions/NavigationTiming  'Update NavigationTiming for schema revision to 7494934' | [production] | 
            
  | 19:59 | <ori> | synchronized php-1.23wmf14/extensions/NavigationTiming  'Update NavigationTiming for schema revision to 7494934' | [production] | 
            
  | 19:55 | <bd808> | during scap test snapshot[1234] reported "sudo: no tty present and no askpass program specified" | [production] | 
            
  | 19:50 | <bd808> | finished scap: no-diff scap to test script changes (duration: 25m 26s) | [production] | 
            
  | 19:29 | <bd808> | no-diff scap updated 366 JSON l10n files | [production] | 
            
  | 19:25 | <bd808> | started scap: no-diff scap to test script changes | [production] | 
            
  | 19:11 | <bd808> | Updating scap on mediawiki-installation dsh hosts | [production] | 
            
  | 19:09 | <aaron> | synchronized wmf-config/mc.php  'Set retry_timeout to -1 for memcached in eqiad only' | [production] | 
            
  | 19:02 | <bd808> | Updated /src/scap on tin to b2d8042 | [production] | 
            
  | 18:29 | <aaron> | synchronized wmf-config/CommonSettings.php  'Set wgMathDisableTexFilter to fix performance regression' | [production] | 
            
  | 18:24 | <bd808> | Starting ganglia-monitor on logstash1001. Filed bug 61384 about problem in elasticsearch_monitoring.py effecting the logstash cluster. | [production] | 
            
  | 17:54 | <bd808> | Running gmond in foreground on logstash1001 to debug elasticsearch reporting | [production] | 
            
  | 17:11 | <bd808> | Restarted ganglia-monitor on logstash1001 to see if that makes the elasticsearch and redis metrics show up in ganglia | [production] | 
            
  | 16:16 | <cmjohnson1> | db1034 swapping cables | [production] | 
            
  | 15:26 | <manybubbles> | aborted reindex due to https://bugzilla.wikimedia.org/show_bug.cgi?id=61377 | [production] | 
            
  | 14:25 | <manybubbles> | beginning cirrus reindex of all wikipedias running cirrus except enwiki | [production] | 
            
  | 14:14 | <springle> | synchronized wmf-config/db-eqiad.php  's1 db1056 full steam' | [production] | 
            
  | 13:55 | <springle> | synchronized wmf-config/db-eqiad.php  's1 repool db1056 warm up' | [production] | 
            
  | 13:39 | <springle> | synchronized wmf-config/db-eqiad.php  's6 pool db1030 depool db1010' | [production] | 
            
  | 13:30 | <apergos> | powercycling ms-be1005, unresponsive even on mgmt console | [production] | 
            
  | 11:49 | <akosiaris> | restarted parsoid on wtp1001 | [production] | 
            
  | 11:47 | <akosiaris> | depooled wtp1004 | [production] | 
            
  | 11:44 | <akosiaris> | restart parsoid on wtp1022. | [production] | 
            
  | 10:50 | <mark> | restarted varnish backend on amssq57 | [production] | 
            
  | 06:08 | <Krinkle> | Reloading Zuul to deploy Ie02531143511f418a6 | [production] | 
            
  | 05:29 | <ori> | synchronized wmf-config/mc.php  'Iac9f51209: Revert 'Set Memcached retry_timeout to -1'' | [production] | 
            
  | 05:28 | <ori> | updated /a/common to {{Gerrit|Iac9f51209}}: Revert "Set Memcached retry_timeout to -1" | [production] | 
            
  | 04:05 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-02-14 04:05:23+00:00 | [production] | 
            
  | 03:43 | <springle> | restarted opendj on virt0 | [production] | 
            
  | 03:20 | <LocalisationUpdate> | completed (1.23wmf14) at 2014-02-14 03:20:37+00:00 | [production] | 
            
  | 03:00 | <mholmquist> | synchronized php-1.23wmf14/extensions/MultimediaViewer/resources/mmv/ui/mmv.ui.metadataPanel.js  'Fix for arrow keys in MultimediaViewer' | [production] | 
            
  | 02:34 | <springle> | synchronized wmf-config/db-eqiad.php  'depool db1056 for pt-table-sync bug 61319' | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.23wmf13) at 2014-02-14 02:27:02+00:00 | [production] | 
            
  | 02:25 | <aaron> | synchronized wmf-config/mc.php  'Set Memcached retry_timeout to -1' | [production] | 
            
  | 01:15 | <springle> | xtrabackup clone db1010 to db1030 | [production] | 
            
  | 01:09 | <ori> | restarting EventLogging on vanadium | [production] | 
            
  | 00:54 | <springle> | synchronized wmf-config/db-eqiad.php  'x1 depool db1030 for maintenance' | [production] | 
            
  | 00:42 | <mflaschen> | synchronized php-1.23wmf14/tests/qunit/suites/resources/mediawiki/mediawiki.jqueryMsg.test.js  'Sync for GENDER fix to jQueryMsg' | [production] | 
            
  | 00:40 | <mflaschen> | synchronized php-1.23wmf14/resources/mediawiki/  'Sync for GENDER fix to jQueryMsg' | [production] | 
            
  | 00:36 | <mflaschen> | synchronized php-1.23wmf13/tests/qunit/suites/resources/mediawiki/mediawiki.jqueryMsg.test.js  'Sync for GENDER fix to jQueryMsg' | [production] | 
            
  | 00:34 | <mflaschen> | synchronized php-1.23wmf13/resources/mediawiki/  'Sync for GENDER fix to jQueryMsg' | [production] | 
            
  | 00:16 | <mholmquist> | synchronized php-1.23wmf14/extensions/MultimediaViewer/resources/mmv/mmv.lightboxinterface.js  'Fix for arrow keys in MultimediaViewer' | [production] | 
            
  | 00:14 | <ebernhardson> | synchronized php-1.23wmf13/extensions/Flow | [production] | 
            
  | 00:07 | <mholmquist> | synchronized php-1.23wmf13/extensions/MultimediaViewer/resources/mmv/mmv.lightboxinterface.js  'Fix for arrow keys in MultimediaViewer' | [production] |