| 
      
        2017-05-24
      
      §
     | 
  
    
  | 08:49 | 
  <akosiaris@puppetmaster1001> | 
  conftool action : set/pooled=no; selector: cp3036.esams.wmnet | 
  [production] | 
            
  | 08:48 | 
  <joal@tin> | 
  Started deploy [analytics/refinery@9377d9c]: Deploying to fix yesterday's deploy bugs | 
  [production] | 
            
  | 07:28 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1094 - T164530 (duration: 00m 41s) | 
  [production] | 
            
  | 07:17 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1086, depool db1094 - T164530 (duration: 00m 41s) | 
  [production] | 
            
  | 07:03 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1079, depool db1086 - T164530 (duration: 00m 42s) | 
  [production] | 
            
  | 06:56 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1079 - T164530 (duration: 00m 54s) | 
  [production] | 
            
  | 06:34 | 
  <marostegui> | 
  Deploy alter table on s2.fawiki directly on codfw master (db2029) after running the clean up duplicates script - https://phabricator.wikimedia.org/T164530 | 
  [production] | 
            
  | 06:04 | 
  <marostegui> | 
  Run pt-table-checksum on s7.frwiktionary - https://phabricator.wikimedia.org/T163190 | 
  [production] | 
            
  | 06:02 | 
  <marostegui> | 
  Deploy alter table on s2  db1047 - https://phabricator.wikimedia.org/T162611 | 
  [production] | 
            
  | 03:04 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Wed May 24 03:04:03 UTC 2017 (duration 6m 45s) | 
  [production] | 
            
  | 02:57 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.30.0-wmf.2) (duration: 13m 38s) | 
  [production] | 
            
  | 02:24 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.30.0-wmf.1) (duration: 07m 30s) | 
  [production] | 
            
  
    | 
      
        2017-05-23
      
      §
     | 
  
    
  | 21:00 | 
  <mepps> | 
  deployed payment wiki 0c06f8e6c998dccaa9857476dd71c16103e9b9e2 | 
  [production] | 
            
  | 20:24 | 
  <bblack> | 
  enable BBR for all caches - T147569 | 
  [production] | 
            
  | 20:20 | 
  <bblack> | 
  enable BBR for all caches @ codfw - T147569 | 
  [production] | 
            
  | 20:10 | 
  <bblack> | 
  enable BBR for all caches @ ulsfo - T147569 | 
  [production] | 
            
  | 20:06 | 
  <bblack> | 
  disabling puppet on all caches for BBR deploy control | 
  [production] | 
            
  | 19:52 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.30.0-wmf.2 | 
  [production] | 
            
  | 19:34 | 
  <thcipriani@tin> | 
  Finished scap: testwiki to php-1.30.0-wmf.2 and rebuild l10n cache (duration: 27m 52s) | 
  [production] | 
            
  | 19:07 | 
  <bblack> | 
  resetting cp1074 queues again: "fq flow_limit 200 buckets 10240" | 
  [production] | 
            
  | 19:06 | 
  <thcipriani@tin> | 
  Started scap: testwiki to php-1.30.0-wmf.2 and rebuild l10n cache | 
  [production] | 
            
  | 18:43 | 
  <bblack> | 
  resetting cp1074 queues again: "fq flow_limit 200 buckets 4096" | 
  [production] | 
            
  | 17:40 | 
  <bblack> | 
  fq on cp1074 reset to flow_limit 200 (resets counters) | 
  [production] | 
            
  | 17:24 | 
  <ladsgroup@tin> | 
  Finished deploy [ores/deploy@4874809]: Trying again with deploying ores (duration: 21m 30s) | 
  [production] | 
            
  | 17:09 | 
  <thcipriani> | 
  starting branch cut for 1.30.0-wmf.2 T163512 | 
  [production] | 
            
  | 17:03 | 
  <ladsgroup@tin> | 
  Started deploy [ores/deploy@4874809]: Trying again with deploying ores | 
  [production] | 
            
  | 16:50 | 
  <volans> | 
  upgrading facter on mw[2250-2259] as a test batch | 
  [production] | 
            
  | 16:49 | 
  <bblack> | 
  BBR: enabling bbr on cp1074 - T147569 | 
  [production] | 
            
  | 16:43 | 
  <bblack> | 
  BBR: enabling mq+fq on cp1074 - T147569 | 
  [production] | 
            
  | 16:26 | 
  <bblack> | 
  puppet re-enables on caches | 
  [production] | 
            
  | 16:24 | 
  <demon@tin> | 
  Synchronized README: testing (duration: 00m 38s) | 
  [production] | 
            
  | 16:17 | 
  <bblack> | 
  disabled puppet on all cp* for RPS-related deployments (just in case!) | 
  [production] | 
            
  | 16:16 | 
  <bblack> | 
  disabled puppet on all lvs* for RPS-related deployments | 
  [production] | 
            
  | 16:15 | 
  <ema> | 
  cp1074: enable prometheus node_exporter qdisc collector T147569 | 
  [production] | 
            
  | 15:50 | 
  <marostegui> | 
  Stop replication on dbstore1002 s7 thread for maintenance - T163190 | 
  [production] | 
            
  | 15:23 | 
  <volans> | 
  re-enabled raid_handler and puppet on tegmen | 
  [production] | 
            
  | 15:02 | 
  <otto@tin> | 
  Finished deploy [eventlogging/analytics@UNKNOWN]: (no justification provided) (duration: 00m 02s) | 
  [production] | 
            
  | 15:01 | 
  <otto@tin> | 
  Started deploy [eventlogging/analytics@UNKNOWN]: (no justification provided) | 
  [production] | 
            
  | 14:56 | 
  <otto@tin> | 
  Finished deploy [eventlogging/analytics@25f8096]: (no justification provided) (duration: 00m 04s) | 
  [production] | 
            
  | 14:56 | 
  <otto@tin> | 
  Started deploy [eventlogging/analytics@25f8096]: (no justification provided) | 
  [production] | 
            
  | 14:42 | 
  <volans> | 
  temporarily disabled raid_handler and puppet on tegmen | 
  [production] | 
            
  | 14:25 | 
  <jynus> | 
  deploying new check_raid monitoring write policy for megacli T166108 | 
  [production] | 
            
  | 14:21 | 
  <Dereckson> | 
  EU SWAT done | 
  [production] | 
            
  | 14:21 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on dty.wikipedia (T166121) (duration: 00m 38s) | 
  [production] | 
            
  | 14:09 | 
  <XioNoX> | 
  re-enabling BGP session to Init7 - T165288 | 
  [production] | 
            
  | 14:03 | 
  <moritzm> | 
  installing nutcracker update in codfw (T163795) | 
  [production] | 
            
  | 13:37 | 
  <marostegui> | 
  Run CleanDuplicateScores script to clean up possible duplicates on fawiki before starting to create the UNIQUE keys - https://phabricator.wikimedia.org/T164530 | 
  [production] | 
            
  | 13:23 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Add *.esa.int to CopyUploadsDomains (T164643) (duration: 00m 39s) | 
  [production] | 
            
  | 12:47 | 
  <elukey@tin> | 
  Finished deploy [analytics/refinery@679aeea]: Updated stat1002 with the last refinery deployment (duration: 00m 42s) | 
  [production] | 
            
  | 12:46 | 
  <elukey@tin> | 
  Started deploy [analytics/refinery@679aeea]: Updated stat1002 with the last refinery deployment | 
  [production] |