| 2019-01-08
      
      ยง | 
    
  | 23:35 | <eileen> | process-control config revision is 9dc6e63fcd | [production] | 
            
  | 23:00 | <XioNoX> | Update pfw3-codfw/eqiad security policies - T213100 | [production] | 
            
  | 22:39 | <XioNoX> | deactivate policy-statement BGP_fundraising_aggregates term nat on pfw3-eqiad/codfw - T211028 | [production] | 
            
  | 22:29 | <gehel> | starting data copy from wdqs1007 to wdqs1008 (both will be depooled) - T213217 | [production] | 
            
  | 22:27 | <jforrester@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: TestCommons: Add default search NSes (duration: 00m 51s) | [production] | 
            
  | 22:22 | <James_F> | Ran /docroot/noc/createTxtFileSymlinks.sh for new dblist | [production] | 
            
  | 22:21 | <jforrester@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Use new wikidatarepo dblist where appropriate (duration: 00m 52s) | [production] | 
            
  | 22:20 | <jforrester@deploy1001> | Synchronized wmf-config/CommonSettings.php: dblists: Load wikibaserepo (duration: 00m 52s) | [production] | 
            
  | 22:15 | <jforrester@deploy1001> | scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) | [production] | 
            
  | 22:14 | <jforrester@deploy1001> | Synchronized dblists/wikidata.dblist: dblists: Remove testcommons from wikidata list (duration: 00m 52s) | [production] | 
            
  | 22:13 | <jforrester@deploy1001> | Synchronized dblists/wikidatarepo.dblist: dblists: Add wikidatarepo list (duration: 00m 53s) | [production] | 
            
  | 22:12 | <urandom> | forcing removal of restbase1016-b (host down way too long to salvage) -- T212418 | [production] | 
            
  | 22:08 | <marostegui> | Drop valid_tag table from db2043 with replication (s3 codfw master - lag will be generated) - T212254 | [production] | 
            
  | 22:03 | <krinkle@deploy1001> | Synchronized wmf-config/CommonSettings.php: cleanup - Idfa129a65a41 (duration: 00m 53s) | [production] | 
            
  | 21:53 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1078 T212254 (duration: 00m 52s) | [production] | 
            
  | 21:49 | <marostegui> | Drop valid_tag table from db1078 (s3) - T212254 | [production] | 
            
  | 21:49 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1078 T212254 (duration: 00m 53s) | [production] | 
            
  | 21:43 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1123 T212254 (duration: 00m 53s) | [production] | 
            
  | 21:38 | <marostegui> | Drop valid_tag table from db1123 (s3) - T212254 | [production] | 
            
  | 21:38 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1123 T212254 (duration: 00m 53s) | [production] | 
            
  | 21:31 | <dduvall@deploy1001> | rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.12 | [production] | 
            
  | 21:03 | <dduvall@deploy1001> | Finished scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache (duration: 39m 22s) | [production] | 
            
  | 20:42 | <ejegg> | updated payments-wiki from b8acb95a2a to c455bbc6bb | [production] | 
            
  | 20:24 | <dduvall@deploy1001> | Started scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache | [production] | 
            
  | 20:24 | <gehel> | starting data copy from wdqs1004 to wdqs1007 (both will be depooled) - T213217 | [production] | 
            
  | 20:21 | <jforrester@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: TestCommons: Don't enable entities, we're not Wikidata.org (duration: 01m 44s) | [production] | 
            
  | 20:11 | <XioNoX> | change BGP_fundraising_aggregates term nat from static to aggregate on pfw3-eqiad - T211028 | [production] | 
            
  | 19:51 | <ejegg> | updated fundraising CiviCRM from b8e3a71845 to 5580f0b11c | [production] | 
            
  | 19:48 | <krinkle@deploy1001> | Finished deploy [performance/navtiming@68fd54d]: (no justification provided) (duration: 00m 05s) | [production] | 
            
  | 19:48 | <krinkle@deploy1001> | Started deploy [performance/navtiming@68fd54d]: (no justification provided) | [production] | 
            
  | 19:48 | <dduvall@deploy1001> | Pruned MediaWiki: 1.33.0-wmf.12 (duration: 06m 26s) | [production] | 
            
  | 19:11 | <arlolra> | Updated Parsoid to 2c5dc7b (T197616, T205491, T209772, T199926, T209194, T204622) | [production] | 
            
  | 19:06 | <marostegui> | Drop valid_tag table from s1 - T212254 | [production] | 
            
  | 19:00 | <arlolra@deploy1001> | Finished deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b (duration: 10m 40s) | [production] | 
            
  | 18:54 | <XioNoX> | make pfw3-codfw source NAT similar to pfw3-eqiad - T211028 | [production] | 
            
  | 18:54 | <ejegg> | updated SmashPig standalone install from fb3268897b to 25713ca232 | [production] | 
            
  | 18:50 | <marostegui> | Drop valid_tag table from s4 - T212254 | [production] | 
            
  | 18:50 | <XioNoX> | add NAT workaround to pfw3-eqiad - T211028 | [production] | 
            
  | 18:49 | <arlolra@deploy1001> | Started deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b | [production] | 
            
  | 18:38 | <XioNoX> | temporarily permit ssh from frpm1001 to pfw3-eqiad on pfw3-eqiad | [production] | 
            
  | 18:28 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T86338 T202167 (duration: 00m 45s) | [production] | 
            
  | 18:27 | <jynus> | restarting s5 replication on labsdb1009/10/11 | [production] | 
            
  | 17:41 | <moritzm> | installing libseccomp updates from stretch point release | [production] | 
            
  | 17:40 | <mobrovac@deploy1001> | Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 (duration: 02m 29s) | [production] | 
            
  | 17:38 | <mobrovac@deploy1001> | Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 | [production] | 
            
  | 17:37 | <mobrovac@deploy1001> | Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - T210752 T197616 (duration: 96m 50s) | [production] | 
            
  | 17:33 | <_joe_> | applying the new apache configuration to jobrunners in eqiad | [production] | 
            
  | 17:24 | <elukey> | roll restart of aqs on aqs100* to pick up new Druid settings | [production] | 
            
  | 17:20 | <_joe_> | depooling mw1299 for testing of the apache change | [production] | 
            
  | 17:16 | <SMalyshev> | restarted Blazegraph wdqs1006 due to unresponsiveness (caused by load?) | [production] |