| 2014-08-21
      
      § | 
    
  | 14:36 | <hashar_> | Jenkins: updating mediawiki code sniffer repo bf82117..bc4e590 | [production] | 
            
  | 10:02 | <hashar> | Jenkins installed plugin [https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin Throttle Concurrent Builds]. | [production] | 
            
  | 03:21 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu Aug 21 03:20:47 UTC 2014 (duration 20m 46s) | [production] | 
            
  | 02:36 | <LocalisationUpdate> | completed (1.24wmf17) at 2014-08-21 02:34:56+00:00 | [production] | 
            
  | 02:20 | <LocalisationUpdate> | completed (1.24wmf16) at 2014-08-21 02:19:26+00:00 | [production] | 
            
  | 00:08 | <MatmaRex> | (manybubbles contd.) …a single node going down but I expect the cluster to stay "yellow" during the process- no alerts. | [production] | 
            
  | 00:07 | <manybubbles> | bd808 needs to plan a logstash upgrade soon - let it be logged | [production] | 
            
  | 00:00 | <manybubbles> | unattended rolling restart of Elasticsearch cluster is going just fine - adding the 30 minute sleep between servers and turning down the replication rate makes it pretty boring. | [production] | 
            
  
    | 2014-08-20
      
      § | 
    
  | 23:07 | <awight> | stopping the Thank You job | [production] | 
            
  | 22:51 | <ori> | disabled puppet on osmium to debug memory leak | [production] | 
            
  | 21:46 | <marktraceur> | Synchronized php-1.24wmf17/extensions/MultimediaViewer/: Add disable-by-default option to MultimediaViewer (duration: 00m 07s) | [production] | 
            
  | 21:10 | <marktraceur> | Synchronized wmf-config: Turn off Media Viewer for logged-in users at Commons. (duration: 00m 07s) | [production] | 
            
  | 21:06 | <marktraceur> | updated /a/common to {{Gerrit|I226bd1468}}: Add item-redirect to OAuth permissions | [production] | 
            
  | 19:50 | <hashar> | Restarting Zuul to prettify build results {{bug|66095}} | [production] | 
            
  | 19:48 | <awight> | Synchronized php-1.24wmf17/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 05s) | [production] | 
            
  | 19:47 | <awight> | Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 04s) | [production] | 
            
  | 19:46 | <awight> | Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 07s) | [production] | 
            
  | 16:11 | <manybubbles> | elastic1001 upgrade went well - upgrading elastic1002 now | [production] | 
            
  | 15:48 | <hashar> | dns: Jenkins will now complain whenever you attempt to send tabs in any file of operations/dns.git {{bug|69478}} | [production] | 
            
  | 15:17 | <manybubbles> | manually lowered elasticsearch recovery speeds to stem off high load caused by healing the restart of elastic1001 - we were slowing down enough that we were filling the pool counter | [production] | 
            
  | 15:05 | <anomie> | Synchronized wmf-config/CommonSettings.php: SWAT: Add item-redirect to OAuth permissions [[gerrit:155257]] (duration: 00m 09s) | [production] | 
            
  | 15:01 | <anomie> | Synchronized php-1.24wmf17/extensions/Wikidata/extensions/Wikibase/lib/: SWAT: Touch files on advice of Wikidata folks (duration: 00m 09s) | [production] | 
            
  | 15:01 | <anomie> | Synchronized wmf-config/Wikibase.php: SWAT: Fix config for specialSiteLinkGroups in Wikibase [[gerrit:155218]] (duration: 00m 09s) | [production] | 
            
  | 14:49 | <manybubbles> | installing elasticsearch 1.3.2 on elasticsearch1001 only right now as a test | [production] | 
            
  | 14:47 | <manybubbles> | upgrading elasticsearch plugins on all elasticsearch servers in preparation to upgrade to elasticsearch 1.3 - if we roll back we'll have to redeploy the plugins | [production] | 
            
  | 14:11 | <ottomata> | changing group ownership and permissions on raw webrequest data in hdfs.  Users now must be in the analytics-privatedata-users group to access. | [production] | 
            
  | 13:47 | <manybubbles> | experimenting with lowering merge factor on enwiki's Cirrus index - should improve query performance at the cost of more background tasks in the Elasticserach cluster | [production] | 
            
  | 13:37 | <ottomata> | disabling puppet on analytics1027 temporarily | [production] | 
            
  | 13:11 | <godog> | reboot ms-be1003, xfs errors/panics | [production] | 
            
  | 12:03 | <ori> | updated /a/common to {{Gerrit|Ic3fe1ef83}}: Update all symlinks to /apache | [production] | 
            
  | 11:36 | <hashar> | Updating Jenkins Job Builder fork 666e953..0268581 | [production] | 
            
  | 11:07 | <hashar_> | mw1019 is missing sync-common causing sync issues. | [production] | 
            
  | 11:06 | <hashar> | Synchronized wmf-config/InitialiseSettings.php: new domain www.veikkos-archiv.com to wgCopyUploadsDomains {{gerrut|155239}} {{bug|69777}} (duration: 00m 03s) | [production] | 
            
  | 11:05 | <hashar> | Synchronized wmf-config/InitialiseSettings.php: new domain www.veikkos-archiv.com to wgCopyUploadsDomains {{gerrut|155239}} {{bug|69777}} (duration: 00m 03s) | [production] | 
            
  | 10:33 | <ori> | Synchronized w/touch.php: Ic9d8837b1: Canonicalize some remaining references to /apache symlink (duration: 00m 05s) | [production] | 
            
  | 10:33 | <ori> | Synchronized w/mobilelanding.php: Ic9d8837b1: Canonicalize some remaining references to /apache symlink (duration: 00m 05s) | [production] | 
            
  | 10:26 | <ori> | updated /a/common to {{Gerrit|Ic9d8837b1}}: Canonicalize some remaining references to /apache symlink | [production] | 
            
  | 10:16 | <ori> | Synchronized wmf-config/CommonSettings.php: Id2d5cfa4c: Canonicalize path to $wgSiteMatrixFile (duration: 00m 06s) | [production] | 
            
  | 09:40 | <godog> | uploaded hhvm_3.3-dev+20140728+wmf5 to carbon | [production] | 
            
  | 09:27 | <hashar> | restarted Jenkins Gearman plugin. | [production] | 
            
  | 04:06 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Aug 20 04:05:41 UTC 2014 (duration 5m 40s) | [production] | 
            
  | 03:12 | <LocalisationUpdate> | completed (1.24wmf17) at 2014-08-20 03:11:08+00:00 | [production] | 
            
  | 02:40 | <LocalisationUpdate> | completed (1.24wmf16) at 2014-08-20 02:39:40+00:00 | [production] | 
            
  
    | 2014-08-19
      
      § | 
    
  | 23:17 | <catrope> | Synchronized php-1.24wmf17/extensions/MobileFrontend: (no message) (duration: 00m 04s) | [production] | 
            
  | 23:15 | <catrope> | Synchronized wmf-config/InitialiseSettings.php: Set wmgWikibaseSiteGroup for wikinews (duration: 00m 05s) | [production] | 
            
  | 22:59 | <aude> | Synchronized php-1.24wmf17/extensions/Wikidata: Fix badges css on Wikidata (duration: 00m 11s) | [production] | 
            
  | 22:30 | <aude> | Finished scap: Update Wikidata, WikimediaMessages and ZeroBanner (duration: 22m 02s) | [production] | 
            
  | 22:08 | <aude> | Started scap: Update Wikidata, WikimediaMessages and ZeroBanner | [production] | 
            
  | 22:03 | <aude> | Synchronized php-1.24wmf17/extensions/ZeroBanner: Update, per yurik (duration: 00m 18s) | [production] | 
            
  | 21:22 | <aude> | Synchronized wikidataclient.dblist: Enable Wikibase on Wikinews (duration: 00m 08s) | [production] |