| 2018-05-01
      
      § | 
    
  | 21:57 | <krinkle@tin> | Synchronized php-1.32.0-wmf.1/includes/libs/rdbms/: Iba663c58224af9 (duration: 01m 19s) | [production] | 
            
  | 21:54 | <awight@tin> | Finished deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627 (duration: 03m 21s) | [production] | 
            
  | 21:50 | <awight@tin> | Started deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627 | [production] | 
            
  | 21:48 | <awight@tin> | Finished deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 (duration: 00m 26s) | [production] | 
            
  | 21:48 | <awight@tin> | Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 | [production] | 
            
  | 21:48 | <awight@tin> | Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 | [production] | 
            
  | 21:44 | <demon@tin> | Finished scap: group0 to wmf.2 (duration: 68m 18s) | [production] | 
            
  | 21:42 | <XioNoX> | re-enabling eqsin-codfw link | [production] | 
            
  | 21:23 | <urandom> | restbase: begin culling leaked revisions, enwiki_T_mobile__ng_{lead,remaining} - T192689 | [production] | 
            
  | 20:43 | <urandom> | restbase: begin culling leaked revisions, commons_T_mobile__ng_remaining - T192689 | [production] | 
            
  | 20:35 | <demon@tin> | Started scap: group0 to wmf.2 | [production] | 
            
  | 20:35 | <ebernhar|lunch> | update stale apifeatureusage-search-svc-eqiad-wmnet template in eqiad elasticsearch and delete unused apifeatureusage template | [production] | 
            
  | 20:28 | <urandom> | restbase: begin culling leaked revisions, commons_T_mobile__ng_lead - T192689 | [production] | 
            
  | 20:06 | <demon@tin> | Pruned MediaWiki: 1.31.0-wmf.27 (duration: 03m 08s) | [production] | 
            
  | 20:00 | <ottomata> | rolling restart of eventbus to apply new logstash formatter version T193230 | [production] | 
            
  | 19:59 | <demon@tin> | Pruned MediaWiki: 1.31.0-wmf.30 [keeping static files] (duration: 01m 43s) | [production] | 
            
  | 19:49 | <urandom> | restbase: begin culling leaked revisions, others_T_mobile__ng_remaining - T192689 | [production] | 
            
  | 19:27 | <ottomata> | rolling restart of eventbus to apply logstash tag https://phabricator.wikimedia.org/T193230 | [production] | 
            
  | 19:17 | <mutante> | mw2174,mw2175,mw2176 ff - reinstalling with wmf-auto-reimage to stretch | [production] | 
            
  | 19:00 | <ottomata> | rolling restart of eventlogging-service-eventbus to apply logstash logging configs - T193230 | [production] | 
            
  | 18:53 | <ariel@tin> | Finished deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone (duration: 00m 04s) | [production] | 
            
  | 18:53 | <ariel@tin> | Started deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone | [production] | 
            
  | 17:40 | <XioNoX> | disabling eqsin<->codfw link for high packet loss on link | [production] | 
            
  | 17:39 | <otto@tin> | Finished deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 (duration: 02m 29s) | [production] | 
            
  | 17:36 | <otto@tin> | Started deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 | [production] | 
            
  | 16:15 | <ebernhardson> | T192972 change eqiad elasticsearch disk watermarks from 85/85 to 80/80 to match disk space alerts | [production] | 
            
  | 16:15 | <herron> | manually kicked off mirror@sodium:~$ /usr/local/sbin/update-ubuntu-mirror to clear ubuntu mirror out of sync alert | [production] | 
            
  | 15:50 | <urandom> | restbase: begin culling leaked revisions, others_T_mobile__ng_lead -- T192689 | [production] | 
            
  | 15:37 | <ejegg> | updated SmashPig standalone from a4de12d415 to 99585c8084 | [production] | 
            
  | 13:35 | <herron> | restarted hhvm on mw1233 | [production] | 
            
  | 02:47 | <l10nupdate@tin> | scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 04s) | [production] | 
            
  | 01:20 | <ejegg> | re-enabled fundraising jobs | [production] | 
            
  | 01:01 | <ejegg> | updated CiviCRM from 47197006d5 to 46883844a3 | [production] | 
            
  | 00:51 | <eileen> | email sent advising of minor interuption | [production] | 
            
  | 00:50 | <ejegg> | disabled fundraising jobs for db update | [production] | 
            
  | 00:21 | <ebernhardson> | increase cluster.routing.allocation.balance.threshold from 1.0 to 1.5 for eqiad elasticsearch cluster to reduce rebalancing agressiveness | [production] | 
            
  
    | 2018-04-30
      
      § | 
    
  | 23:38 | <catrope@tin> | Synchronized php-1.32.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.DiffPage.init.js: T192755 (duration: 00m 59s) | [production] | 
            
  | 23:36 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Set $wgKartographerUsePageLanguage to false everywhere (T192955) (duration: 00m 59s) | [production] | 
            
  | 23:33 | <catrope@tin> | Synchronized php-1.32.0-wmf.1/extensions/AbuseFilter/includes/AbuseFilter.php: Fix notices when disallowing edits (duration: 00m 59s) | [production] | 
            
  | 23:21 | <catrope@tin> | Synchronized wmf-config/: USe internal cluster for SPARQL services (T192942) (duration: 01m 02s) | [production] | 
            
  | 23:14 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Config cleanup patches from SWAT (duration: 01m 00s) | [production] | 
            
  | 23:05 | <mutante> | ores1001: rm -rf /srv/deployment/ores/venv/ (T193422) | [production] | 
            
  | 21:46 | <ebernhardson> | T192972 increase eqiad elasticsearch disk watermarks from 75/80 to 85/85 | [production] | 
            
  | 20:27 | <arlolra> | Updated Parsoid to 50b0588 (T186358, T191700, T192909) | [production] | 
            
  | 20:22 | <awight@tin> | Started deploy [ores/deploy@bf182e2]: ORES: Include bot edits in precaching wikidata itemquality; T187927 | [production] | 
            
  | 20:21 | <arlolra@tin> | Finished deploy [parsoid/deploy@d8d7b42]: Updating Parsoid to 50b0588 (duration: 09m 46s) | [production] | 
            
  | 20:20 | <awight@tin> | Finished deploy [ores/deploy@5b27205]: Rollback ores1001 to master (duration: 02m 56s) | [production] | 
            
  | 20:19 | <bsitzmann@tin> | Finished deploy [mobileapps/deploy@d3724d2]: Update mobileapps to cc00cae (T191869) (duration: 07m 32s) | [production] | 
            
  | 20:17 | <awight@tin> | Started deploy [ores/deploy@5b27205]: Rollback ores1001 to master | [production] | 
            
  | 20:11 | <bsitzmann@tin> | Started deploy [mobileapps/deploy@d3724d2]: Update mobileapps to cc00cae (T191869) | [production] |