| 2016-05-16
      
      § | 
    
  | 23:09 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 36s) | [production] | 
            
  | 23:08 | <catrope@tin> | Synchronized wmf-config/CirrusSearch-common.php: SWAT (duration: 00m 25s) | [production] | 
            
  | 17:06 | <ejegg> | updated payments from 1af7208b4c7387a701f9ce83e33483d1e19213cb to 0b2d6807f68e7ccfc36a431e90d563b4d5a9e966 | [production] | 
            
  | 17:00 | <godog> | bounce cassandra-b on restbase2008-b to restart bootstrap | [production] | 
            
  | 16:59 | <ottomata> | restarting kafka broker on kafka1022 | [production] | 
            
  | 16:54 | <ottomata> | restarted kafka broker on kafka1020 (about 5 mins ago) | [production] | 
            
  | 16:36 | <ottomata> | restarting kafka broker on kafka1018 | [production] | 
            
  | 16:23 | <ottomata> | restarting kafka broker on kafka1014 | [production] | 
            
  | 16:15 | <ottomata> | restarting broker on kafka1013 | [production] | 
            
  | 15:42 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Add an alias "ΒΠ" for Project namespace in elwiki [[gerrit:288965]] (duration: 00m 26s) | [production] | 
            
  | 15:36 | <thcipriani@tin> | Synchronized wmf-config/CommonSettings.php: SWAT: Enable MathML rendering by default on test, wikidata and dewikibooks PART II [[gerrit:286180]] (duration: 00m 24s) | [production] | 
            
  | 15:35 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable MathML rendering by default on test, wikidata and dewikibooks PART I [[gerrit:286180]] (duration: 00m 26s) | [production] | 
            
  | 15:31 | <bblack> | starting slow cache_upload frontend restarts (wipes) for cache size upgrades (~10H process) | [production] | 
            
  | 15:29 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Add "deletedtext" right to "eliminator" user group on fawiki [[gerrit:288956]] (duration: 00m 26s) | [production] | 
            
  | 15:25 | <thcipriani@tin> | Synchronized wmf-config/abusefilter.php: SWAT: Enable blocking feature of AbuseFilter on ptwiktionary [[gerrit:288810]] (duration: 00m 24s) | [production] | 
            
  | 15:14 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Set meta namespace names for jamwiki [[gerrit:288628]] (duration: 00m 27s) | [production] | 
            
  | 15:09 | <bblack> | varnish package upgrade on cache_maps done | [production] | 
            
  | 15:08 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: wgCopyUploadProxy: Vary per datacenter [[gerrit:287095]] (duration: 00m 26s) | [production] | 
            
  | 15:01 | <hoo@tin> | Synchronized wmf-config/Wikibase.php: Update Wikidata property blacklist (duration: 00m 25s) | [production] | 
            
  | 14:55 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Remove all mentions to db1027, db2008 and db2009 (duration: 00m 33s) | [production] | 
            
  | 14:54 | <jynus@tin> | Synchronized wmf-config/db-codfw.php: Remove all mentions to db1027, db2008 and db2009 (duration: 00m 34s) | [production] | 
            
  | 14:45 | <mobrovac> | change-propagation deployed ef5f6ff55 | [production] | 
            
  | 14:45 | <bblack> | starting varnish package upgrade on cache_maps | [production] | 
            
  | 14:39 | <bblack> | general package updates on cache clusters (4.4.0 image, libtasn6-1, libjansson4, libidn11) | [production] | 
            
  | 11:20 | <jynus> | testing create table on s7-master T130700 | [production] | 
            
  | 10:08 | <jynus> | applying puppet and restarting sanitarium instances to apply new filter (temporary labs lag) | [production] | 
            
  | 09:37 | <volans> | Slowly re-enabling Puppet on db*, es*, pc*, labsdb*, labservices*, silver, holmium after merged change 288361 T133780 | [production] | 
            
  | 09:31 | <volans> | restarted mysql on db2034 to test merged change 288361 , T133780 | [production] | 
            
  
    | 2016-05-13
      
      § | 
    
  | 23:54 | <catrope@tin> | Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: Do not reuse CentralAuth tokens (duration: 00m 25s) | [production] | 
            
  | 22:54 | <catrope@tin> | Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: Logging live hack (duration: 00m 31s) | [production] | 
            
  | 22:50 | <catrope@tin> | Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: More logging (duration: 00m 25s) | [production] | 
            
  | 22:38 | <catrope@tin> | Synchronized php-1.28.0-wmf.1/extensions/Flow/includes/Notifications/Formatter.php: Fix fatal for old Flow notifications (duration: 00m 26s) | [production] | 
            
  | 22:34 | <catrope@tin> | Synchronized php-1.28.0-wmf.1/extensions/Echo/: Log failures to fetch cross-wiki notifications (duration: 00m 41s) | [production] | 
            
  | 21:55 | <catrope@tin> | Synchronized php-1.28.0-wmf.1/extensions/Echo/Echo.php: Bump cache version now that cache pollution is hopefully fixed (duration: 00m 25s) | [production] | 
            
  | 21:52 | <catrope@tin> | Synchronized php-1.28.0-wmf.1/extensions/Echo/: Fixes for cross-wiki notifications deployment fallout (duration: 00m 38s) | [production] | 
            
  | 21:17 | <bblack> | cache_misc varnish3 downgrade complete (except 3007 + 1045 - those remain depooled/downtimed/puppet-disabled/etc, do not touch) | [production] | 
            
  | 21:06 | <catrope@tin> | Synchronized wmf-config/CommonSettings.php: Enable $wgEchoCrossWikiNotifications on the right wikis (unused for now) (duration: 00m 28s) | [production] | 
            
  | 20:09 | <mattflaschen@tin> | Synchronized wmf-config: Disable cross-wiki notifications entirely on non-SUL wikis and hide preference (duration: 00m 34s) | [production] | 
            
  | 20:05 | <bblack> | starting varnish3 downgrade process for most of cache_misc | [production] | 
            
  | 17:43 | <godog> | powercycle ms-be2008, sdb failed | [production] | 
            
  | 15:41 | <godog> | bounce graphite-web on graphite1001, debugging carbon-c-relay stalls while sending metrics | [production] | 
            
  | 13:57 | <jynus> | installing and testing sqlproxy on dbproxy1005 | [production] | 
            
  | 13:44 | <hashar> | Jenkins zmq is all fine | [production] | 
            
  | 13:43 | <hashar> | Restarted Jenkins, the zmq plugin daemon did not start | [production] | 
            
  | 13:13 | <ema> | depooling cp3007 to try reproducing T134989 | [production] | 
            
  | 11:28 | <ema> | repooling cp1061 | [production] |