| 2018-05-23
      
      § | 
    
  | 10:21 | <jynus> | restarting db1117 for setup and upgrade | [production] | 
            
  | 10:17 | <dcausse> | manually updating mapping on wikidatawiki elastic indices to add new lexeme fields | [production] | 
            
  | 10:15 | <addshore> | updateSearchIndexConfig.php with --justMapping failed :( | [production] | 
            
  | 10:15 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 01m 20s) | [production] | 
            
  | 10:14 | <addshore> | addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki wikidatawiki --justMapping --cluster codfw | [production] | 
            
  | 09:35 | <addshore> | Finished, forceSearchIndex.php --wiki testwikidatawiki | [production] | 
            
  | 09:34 | <moritzm> | upgrading remaining job runners in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 09:30 | <marostegui> | Deploy schema change on db1101:3318 - T191519 T188299 T190148 T194270 | [production] | 
            
  | 09:30 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 01m 20s) | [production] | 
            
  | 09:03 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 01m 19s) | [production] | 
            
  | 09:02 | <addshore> | addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki testwikidatawiki --queue | [production] | 
            
  | 09:00 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: [[gerrit:434642|[testwikidata] Add Lexeme NS to ContentNamespaces]] (duration: 01m 20s) | [production] | 
            
  | 08:31 | <moritzm> | upgrading remaining app/API servers in codfw to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 08:21 | <marostegui> | Deploy schema change on db1099:3318 - T191519 T188299 T190148 T194270 | [production] | 
            
  | 08:20 | <addshore> | all the updateSearchIndexConfig runs from wasat also had --cluster=codfw | [production] | 
            
  | 08:20 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 01m 20s) | [production] | 
            
  | 08:18 | <addshore> | addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now | [production] | 
            
  | 08:18 | <addshore> | addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now | [production] | 
            
  | 08:17 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: [[gerrit:434592|testwikidata: Add Lexeme NS to wgNamespacesToBeSearchedDefault]] (duration: 01m 19s) | [production] | 
            
  | 08:06 | <addshore> | addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now --reindexProcesses=10 | [production] | 
            
  | 08:00 | <moritzm> | upgrading remaining app servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 07:54 | <addshore> | addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now | [production] | 
            
  | 07:52 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: [[gerrit:434591|testwikidata: Add Property NS to wgNamespacesToBeSearchedDefault]] (duration: 01m 20s) | [production] | 
            
  | 07:45 | <addshore@tin> | Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme: [[gerrit:434602|Use ISO code und for missing language code]] (duration: 01m 30s) | [production] | 
            
  | 07:44 | <addshore@tin> | Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: [[gerrit:434601|Use ISO code und for missing language code]] (duration: 01m 31s) | [production] | 
            
  | 07:38 | <moritzm> | upgrading remaining API servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 07:29 | <elukey> | upload druid debs 0.11.0-3 to stretch-wikimedia | [production] | 
            
  | 06:52 | <moritzm> | upgrading mw1299-mw1306 (job runners) to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 | [production] | 
            
  | 06:49 | <marostegui> | Re-add indexes on wb_terms on db1092 - T194273 | [production] | 
            
  | 06:44 | <elukey> | restart zookeeper on druid100[4-6] for openjdk-8 upgrades | [production] | 
            
  | 06:44 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s) | [production] | 
            
  | 06:22 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 20s) | [production] | 
            
  | 05:43 | <marostegui> | Deploy schema change on db1092 - T191519 T188299 T190148 T194273 T194270 | [production] | 
            
  | 05:43 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 27s) | [production] | 
            
  | 05:21 | <marostegui> | Deploy schema change on dbstore1002:s8 - T191519 T188299 T190148 T194273 T194270 | [production] | 
            
  | 05:12 | <marostegui> | Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194273 | [production] | 
            
  | 05:06 | <marostegui> | Deploy schema change on s3 primary master (db1075) - T191519 T188299 T190148 | [production] | 
            
  | 04:04 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Wed May 23 04:04:43 UTC 2018 (duration 7m 22s) | [production] | 
            
  | 03:57 | <l10nupdate@tin> | scap sync-l10n completed (1.32.0-wmf.5) (duration: 16m 39s) | [production] | 
            
  | 02:58 | <l10nupdate@tin> | scap sync-l10n completed (1.32.0-wmf.4) (duration: 10m 54s) | [production] | 
            
  
    | 2018-05-22
      
      § | 
    
  | 23:29 | <maxsem@tin> | Synchronized dblists/categories-rdf.dblist: https://gerrit.wikimedia.org/r/#/c/433740/ (duration: 01m 17s) | [production] | 
            
  | 22:59 | <twentyafterfour> | train for group0 1.32.0-wmf.5 completed. Tune in tomorrow for more excitement! | [production] | 
            
  | 22:58 | <twentyafterfour@tin> | rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.5  refs T191051 | [production] | 
            
  | 22:54 | <twentyafterfour@tin> | Finished scap: testwikis wikis to 1.32.0-wmf.5  refs T191051 (duration: 76m 01s) | [production] | 
            
  | 21:38 | <twentyafterfour@tin> | Started scap: testwikis wikis to 1.32.0-wmf.5  refs T191051 | [production] | 
            
  | 21:13 | <twentyafterfour@tin> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2833418486" --threads=10 --lang en  --quiet' returned non-zero exit status 255 (duration: 39m 54s) | [production] | 
            
  | 21:12 | <ejegg> | updated CiviCRM from 5b8c868a00 to 4d797fc592 | [production] | 
            
  | 20:33 | <twentyafterfour@tin> | Started scap: testwikis wikis to 1.32.0-wmf.5  refs T191051 | [production] | 
            
  | 19:45 | <pnorman@tin> | Finished deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution (duration: 02m 46s) | [production] | 
            
  | 19:43 | <pnorman@tin> | Started deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution | [production] |