| 
      
        2019-01-08
      
      ยง
     | 
  
    
  | 22:20 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: dblists: Load wikibaserepo (duration: 00m 52s) | 
  [production] | 
            
  | 22:15 | 
  <jforrester@deploy1001> | 
  scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) | 
  [production] | 
            
  | 22:14 | 
  <jforrester@deploy1001> | 
  Synchronized dblists/wikidata.dblist: dblists: Remove testcommons from wikidata list (duration: 00m 52s) | 
  [production] | 
            
  | 22:13 | 
  <jforrester@deploy1001> | 
  Synchronized dblists/wikidatarepo.dblist: dblists: Add wikidatarepo list (duration: 00m 53s) | 
  [production] | 
            
  | 22:12 | 
  <urandom> | 
  forcing removal of restbase1016-b (host down way too long to salvage) -- T212418 | 
  [production] | 
            
  | 22:08 | 
  <marostegui> | 
  Drop valid_tag table from db2043 with replication (s3 codfw master - lag will be generated) - T212254 | 
  [production] | 
            
  | 22:03 | 
  <krinkle@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: cleanup - Idfa129a65a41 (duration: 00m 53s) | 
  [production] | 
            
  | 21:53 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1078 T212254 (duration: 00m 52s) | 
  [production] | 
            
  | 21:49 | 
  <marostegui> | 
  Drop valid_tag table from db1078 (s3) - T212254 | 
  [production] | 
            
  | 21:49 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1078 T212254 (duration: 00m 53s) | 
  [production] | 
            
  | 21:43 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1123 T212254 (duration: 00m 53s) | 
  [production] | 
            
  | 21:38 | 
  <marostegui> | 
  Drop valid_tag table from db1123 (s3) - T212254 | 
  [production] | 
            
  | 21:38 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1123 T212254 (duration: 00m 53s) | 
  [production] | 
            
  | 21:31 | 
  <dduvall@deploy1001> | 
  rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.12 | 
  [production] | 
            
  | 21:03 | 
  <dduvall@deploy1001> | 
  Finished scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache (duration: 39m 22s) | 
  [production] | 
            
  | 20:42 | 
  <ejegg> | 
  updated payments-wiki from b8acb95a2a to c455bbc6bb | 
  [production] | 
            
  | 20:24 | 
  <dduvall@deploy1001> | 
  Started scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache | 
  [production] | 
            
  | 20:24 | 
  <gehel> | 
  starting data copy from wdqs1004 to wdqs1007 (both will be depooled) - T213217 | 
  [production] | 
            
  | 20:21 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: TestCommons: Don't enable entities, we're not Wikidata.org (duration: 01m 44s) | 
  [production] | 
            
  | 20:11 | 
  <XioNoX> | 
  change BGP_fundraising_aggregates term nat from static to aggregate on pfw3-eqiad - T211028 | 
  [production] | 
            
  | 19:51 | 
  <ejegg> | 
  updated fundraising CiviCRM from b8e3a71845 to 5580f0b11c | 
  [production] | 
            
  | 19:48 | 
  <krinkle@deploy1001> | 
  Finished deploy [performance/navtiming@68fd54d]: (no justification provided) (duration: 00m 05s) | 
  [production] | 
            
  | 19:48 | 
  <krinkle@deploy1001> | 
  Started deploy [performance/navtiming@68fd54d]: (no justification provided) | 
  [production] | 
            
  | 19:48 | 
  <dduvall@deploy1001> | 
  Pruned MediaWiki: 1.33.0-wmf.12 (duration: 06m 26s) | 
  [production] | 
            
  | 19:11 | 
  <arlolra> | 
  Updated Parsoid to 2c5dc7b (T197616, T205491, T209772, T199926, T209194, T204622) | 
  [production] | 
            
  | 19:06 | 
  <marostegui> | 
  Drop valid_tag table from s1 - T212254 | 
  [production] | 
            
  | 19:00 | 
  <arlolra@deploy1001> | 
  Finished deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b (duration: 10m 40s) | 
  [production] | 
            
  | 18:54 | 
  <XioNoX> | 
  make pfw3-codfw source NAT similar to pfw3-eqiad - T211028 | 
  [production] | 
            
  | 18:54 | 
  <ejegg> | 
  updated SmashPig standalone install from fb3268897b to 25713ca232 | 
  [production] | 
            
  | 18:50 | 
  <marostegui> | 
  Drop valid_tag table from s4 - T212254 | 
  [production] | 
            
  | 18:50 | 
  <XioNoX> | 
  add NAT workaround to pfw3-eqiad - T211028 | 
  [production] | 
            
  | 18:49 | 
  <arlolra@deploy1001> | 
  Started deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b | 
  [production] | 
            
  | 18:38 | 
  <XioNoX> | 
  temporarily permit ssh from frpm1001 to pfw3-eqiad on pfw3-eqiad | 
  [production] | 
            
  | 18:28 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T86338 T202167 (duration: 00m 45s) | 
  [production] | 
            
  | 18:27 | 
  <jynus> | 
  restarting s5 replication on labsdb1009/10/11 | 
  [production] | 
            
  | 17:41 | 
  <moritzm> | 
  installing libseccomp updates from stretch point release | 
  [production] | 
            
  | 17:40 | 
  <mobrovac@deploy1001> | 
  Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 (duration: 02m 29s) | 
  [production] | 
            
  | 17:38 | 
  <mobrovac@deploy1001> | 
  Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 | 
  [production] | 
            
  | 17:37 | 
  <mobrovac@deploy1001> | 
  Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - T210752 T197616 (duration: 96m 50s) | 
  [production] | 
            
  | 17:33 | 
  <_joe_> | 
  applying the new apache configuration to jobrunners in eqiad | 
  [production] | 
            
  | 17:24 | 
  <elukey> | 
  roll restart of aqs on aqs100* to pick up new Druid settings | 
  [production] | 
            
  | 17:20 | 
  <_joe_> | 
  depooling mw1299 for testing of the apache change | 
  [production] | 
            
  | 17:16 | 
  <SMalyshev> | 
  restarted Blazegraph wdqs1006 due to unresponsiveness (caused by load?) | 
  [production] | 
            
  | 16:56 | 
  <urandom> | 
  forcing removal of restbase1016-a (host down way too long to salvage) -- T212418 | 
  [production] | 
            
  | 16:56 | 
  <jynus> | 
  changing db1124:s5 replication to db2066 | 
  [production] | 
            
  | 16:54 | 
  <marostegui> | 
  Deploy schema change on db1105:3311 T86338 T202167 | 
  [production] | 
            
  | 16:54 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T86338 T202167 (duration: 00m 44s) | 
  [production] | 
            
  | 16:54 | 
  <jynus> | 
  stopping s5 replication on labsdb1009/10/11 to prevent undoable mistakes | 
  [production] | 
            
  | 16:34 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Repool es2019 - T212833 (duration: 02m 51s) | 
  [production] | 
            
  | 16:29 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1089 T86338 T202167 (duration: 00m 45s) | 
  [production] |