| 2018-05-23
      
      § | 
    
  | 08:18 | <addshore> | addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now | [production] | 
            
  | 08:17 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: [[gerrit:434592|testwikidata: Add Lexeme NS to wgNamespacesToBeSearchedDefault]] (duration: 01m 19s) | [production] | 
            
  | 08:06 | <addshore> | addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now --reindexProcesses=10 | [production] | 
            
  | 08:00 | <moritzm> | upgrading remaining app servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 07:54 | <addshore> | addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now | [production] | 
            
  | 07:52 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: [[gerrit:434591|testwikidata: Add Property NS to wgNamespacesToBeSearchedDefault]] (duration: 01m 20s) | [production] | 
            
  | 07:45 | <addshore@tin> | Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme: [[gerrit:434602|Use ISO code und for missing language code]] (duration: 01m 30s) | [production] | 
            
  | 07:44 | <addshore@tin> | Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: [[gerrit:434601|Use ISO code und for missing language code]] (duration: 01m 31s) | [production] | 
            
  | 07:38 | <moritzm> | upgrading remaining API servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 07:29 | <elukey> | upload druid debs 0.11.0-3 to stretch-wikimedia | [production] | 
            
  | 06:52 | <moritzm> | upgrading mw1299-mw1306 (job runners) to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 06:49 | <marostegui> | Re-add indexes on wb_terms on db1092 - T194273 | [production] | 
            
  | 06:44 | <elukey> | restart zookeeper on druid100[4-6] for openjdk-8 upgrades | [production] | 
            
  | 06:44 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s) | [production] | 
            
  | 06:22 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 20s) | [production] | 
            
  | 05:43 | <marostegui> | Deploy schema change on db1092 - T191519 T188299 T190148 T194273 T194270 | [production] | 
            
  | 05:43 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 27s) | [production] | 
            
  | 05:21 | <marostegui> | Deploy schema change on dbstore1002:s8 - T191519 T188299 T190148 T194273 T194270 | [production] | 
            
  | 05:12 | <marostegui> | Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194273 | [production] | 
            
  | 05:06 | <marostegui> | Deploy schema change on s3 primary master (db1075) - T191519 T188299 T190148 | [production] | 
            
  | 04:04 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Wed May 23 04:04:43 UTC 2018 (duration 7m 22s) | [production] | 
            
  | 03:57 | <l10nupdate@tin> | scap sync-l10n completed (1.32.0-wmf.5) (duration: 16m 39s) | [production] | 
            
  | 02:58 | <l10nupdate@tin> | scap sync-l10n completed (1.32.0-wmf.4) (duration: 10m 54s) | [production] | 
            
  
    | 2018-05-22
      
      § | 
    
  | 23:29 | <maxsem@tin> | Synchronized dblists/categories-rdf.dblist: https://gerrit.wikimedia.org/r/#/c/433740/ (duration: 01m 17s) | [production] | 
            
  | 22:59 | <twentyafterfour> | train for group0 1.32.0-wmf.5 completed. Tune in tomorrow for more excitement! | [production] | 
            
  | 22:58 | <twentyafterfour@tin> | rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.5  refs T191051 | [production] | 
            
  | 22:54 | <twentyafterfour@tin> | Finished scap: testwikis wikis to 1.32.0-wmf.5  refs T191051 (duration: 76m 01s) | [production] | 
            
  | 21:38 | <twentyafterfour@tin> | Started scap: testwikis wikis to 1.32.0-wmf.5  refs T191051 | [production] | 
            
  | 21:13 | <twentyafterfour@tin> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2833418486" --threads=10 --lang en  --quiet' returned non-zero exit status 255 (duration: 39m 54s) | [production] | 
            
  | 21:12 | <ejegg> | updated CiviCRM from 5b8c868a00 to 4d797fc592 | [production] | 
            
  | 20:33 | <twentyafterfour@tin> | Started scap: testwikis wikis to 1.32.0-wmf.5  refs T191051 | [production] | 
            
  | 19:45 | <pnorman@tin> | Finished deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution (duration: 02m 46s) | [production] | 
            
  | 19:43 | <pnorman@tin> | Started deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution | [production] | 
            
  | 19:39 | <pnorman@tin> | Finished deploy [tilerator/deploy@18faaa6]: (no justification provided) (duration: 00m 29s) | [production] | 
            
  | 19:38 | <pnorman@tin> | Started deploy [tilerator/deploy@18faaa6]: (no justification provided) | [production] | 
            
  | 19:35 | <pnorman@tin> | Finished deploy [tilerator/deploy@18faaa6]: (no justification provided) (duration: 00m 25s) | [production] | 
            
  | 19:35 | <pnorman@tin> | Started deploy [tilerator/deploy@18faaa6]: (no justification provided) | [production] | 
            
  | 19:34 | <pnorman@tin> | Finished deploy [tilerator/deploy@a4a3fc7]: (no justification provided) (duration: 46m 19s) | [production] | 
            
  | 19:17 | <twentyafterfour@tin> | Synchronized php-1.32.0-wmf.4/extensions/ContentTranslation/: sync https://gerrit.wikimedia.org/r/#/c/434529/ to fix T194810 (duration: 01m 02s) | [production] | 
            
  | 19:14 | <twentyafterfour@tin> | rebuilt and synchronized wikiversions files: group2 wikis to 1.32.0-wmf.4  refs T191050 | [production] | 
            
  | 19:03 | <twentyafterfour> | Today's train: Promoting group2 wikis to 1.32.0-wmf.4 followed by group0 to 1.32.0-wmf.5 | [production] | 
            
  | 18:48 | <pnorman@tin> | Started deploy [tilerator/deploy@a4a3fc7]: (no justification provided) | [production] | 
            
  | 18:44 | <pnorman> | tilerator deploying a4a3fc7b2694a469cc26a6d0cc440d4d46fa9485 | [production] | 
            
  | 17:03 | <marostegui> | Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194270 | [production] | 
            
  | 16:52 | <elukey> | restart zookeeper on druid100[1,3] to complete the openjdk-8 upgrade | [production] | 
            
  | 16:51 | <addshore> | addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now | [production] | 
            
  | 16:43 | <elukey> | upload druid debs 0.11.0-3 to jessie-wikimedia | [production] | 
            
  | 16:25 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Remove db2064 from config - T195228 (duration: 01m 18s) | [production] | 
            
  | 16:24 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Remove db2064 from config - T195228 (duration: 01m 19s) | [production] | 
            
  | 16:14 | <moritzm> | upgrading remaining video scalers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] |