| 
      
        2017-05-10
      
      §
     | 
  
    
  | 08:59 | 
  <moritzm> | 
  installing wget security updates on jessie | 
  [production] | 
            
  | 08:35 | 
  <moritzm> | 
  rebooting mx2001 for update to Linux 4.9 | 
  [production] | 
            
  | 08:35 | 
  <hashar@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: wmgUseTwoColConflict true for all wikis (duration: 00m 54s) | 
  [production] | 
            
  | 07:30 | 
  <marostegui> | 
  Stop replication at the same position on db10418 and db2017 - T147166 https://phabricator.wikimedia.org/T130067 | 
  [production] | 
            
  | 07:16 | 
  <marostegui> | 
  Disable replication codfw > eqiad on s2 -T147166 T130067 | 
  [production] | 
            
  | 07:13 | 
  <Amir1> | 
  another round of cleaning up ores_classification is done, 12M rows deleted. Current number of rows: 64,902,521 (T159753) | 
  [production] | 
            
  | 06:36 | 
  <moritzm> | 
  installing rtmpdump security updates on trusty | 
  [production] | 
            
  | 06:15 | 
  <marostegui> | 
  Deploy alter table wikidatawiki.wb_terms on dbstore1001 - T162539 T163190 | 
  [production] | 
            
  | 06:08 | 
  <marostegui> | 
  Run pt-table-checksum on s7.frwiktionary - T163190 | 
  [production] | 
            
  | 05:04 | 
  <Amir1> | 
  start of cleaning up ores_classification rows for three hours | 
  [production] | 
            
  | 04:49 | 
  <kartik@tin> | 
  Finished deploy [cxserver/deploy@533b4f4]: Update cxserver to 534619c (duration: 02m 38s) | 
  [production] | 
            
  | 04:46 | 
  <kartik@tin> | 
  Started deploy [cxserver/deploy@533b4f4]: Update cxserver to 534619c | 
  [production] | 
            
  | 03:02 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Wed May 10 03:02:23 UTC 2017 (duration 6m 37s) | 
  [production] | 
            
  | 02:55 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.30.0-wmf.1) (duration: 06m 50s) | 
  [production] | 
            
  | 02:30 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.29.0-wmf.21) (duration: 08m 09s) | 
  [production] | 
            
  | 00:29 | 
  <maxsem@tin> | 
  Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/352980/3 (duration: 00m 42s) | 
  [production] | 
            
  | 00:12 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable ORES on fiwiki (T163011) (duration: 00m 43s) | 
  [production] | 
            
  | 00:10 | 
  <RoanKattouw> | 
  Running extensions/ORES/maintenance/PopulateDatabase.php on fiwiki | 
  [production] | 
            
  
    | 
      
        2017-05-09
      
      §
     | 
  
    
  | 23:54 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters beta feature on all remaining wikis (T144458) (duration: 00m 44s) | 
  [production] | 
            
  | 23:33 | 
  <mutante> | 
  db1040 - remove from puppet, puppet node clean/deactivate, deleted salt-key, remove from icinga by running puppet on tegmen after that (T164057) | 
  [production] | 
            
  | 23:23 | 
  <demon@tin> | 
  Finished scap: rebuilding l10n for extension-list swap (duration: 34m 10s) | 
  [production] | 
            
  | 23:13 | 
  <mutante> | 
  analytics1027 - decom: revoke puppet cert, delete salt key, puppet node clean/deactivate, check icinga removal (T161597) | 
  [production] | 
            
  | 22:49 | 
  <demon@tin> | 
  Started scap: rebuilding l10n for extension-list swap | 
  [production] | 
            
  | 22:46 | 
  <reedy@tin> | 
  Synchronized wmf-config/extension-list-wikitech: Consistency (duration: 00m 42s) | 
  [production] | 
            
  | 22:20 | 
  <reedy@tin> | 
  Synchronized wmf-config/wikitech.php: Disable Semantic extensions (duration: 00m 42s) | 
  [production] | 
            
  | 22:03 | 
  <reedy@tin> | 
  scap aborted: (no justification provided) (duration: 00m 03s) | 
  [production] | 
            
  | 22:03 | 
  <reedy@tin> | 
  Started scap: (no justification provided) | 
  [production] | 
            
  | 21:40 | 
  <twentyafterfour> | 
  Mediawiki train group0 finished, will resume tomorrow with group 1 wikis. refs T162954 | 
  [production] | 
            
  | 21:32 | 
  <twentyafterfour> | 
  group0 wikis to 1.30.0-wmf.1 refs T162954 | 
  [production] | 
            
  | 21:32 | 
  <twentyafterfour@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.1 | 
  [production] | 
            
  | 20:52 | 
  <twentyafterfour@tin> | 
  Finished scap: MediaWiki sync new branch wmf/1.30.0-wmf.1 + localization cache and deploy to testwikis refs T162954 (duration: 29m 41s) | 
  [production] | 
            
  | 20:22 | 
  <twentyafterfour@tin> | 
  Started scap: MediaWiki sync new branch wmf/1.30.0-wmf.1 + localization cache and deploy to testwikis refs T162954 | 
  [production] | 
            
  | 19:47 | 
  <maxsem@tin> | 
  Finished deploy [kartotherian/deploy@740235c]: https://gerrit.wikimedia.org/r/#/c/352886/ (duration: 05m 35s) | 
  [production] | 
            
  | 19:42 | 
  <maxsem@tin> | 
  Started deploy [kartotherian/deploy@740235c]: https://gerrit.wikimedia.org/r/#/c/352886/ | 
  [production] | 
            
  | 18:39 | 
  <bblack> | 
  varnish: manually etting runtime lru_interval / nuke_limit via varnishadm for all clusters' backends to match start-time change in https://gerrit.wikimedia.org/r/#/c/352827/ | 
  [production] | 
            
  | 18:26 | 
  <subbu> | 
  updated Parsoid to 9d8badc8 (T151277) | 
  [production] | 
            
  | 18:22 | 
  <ssastry@tin> | 
  Finished deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 (duration: 07m 09s) | 
  [production] | 
            
  | 18:16 | 
  <mepps> | 
  updated SmashPig  from 200f63eedb05f5e6665d9837bba97e7e7237a41d to 0145e2dd3a5c69d61f0dd9b210f6fddc56dc6aec | 
  [production] | 
            
  | 18:15 | 
  <ssastry@tin> | 
  Started deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 | 
  [production] | 
            
  | 17:29 | 
  <elukey> | 
  executing varnish-backend-restart on cp1072 as attempt to mitigate "FetchError Could not get storage" and "ExpKill LRU_Fail" - T145661 | 
  [production] | 
            
  | 17:25 | 
  <elukey> | 
  executing varnish-backend-restart on cp1074 as attempt to mitigate "FetchError Could not get storage" and "ExpKill LRU_Fail" - T145661 | 
  [production] | 
            
  | 17:23 | 
  <twentyafterfour> | 
  Preparing to branch 1.30.0-wmf.1 [ T162954 ] | 
  [production] | 
            
  | 16:08 | 
  <elukey> | 
  playing with mw2146 for T163674 | 
  [production] | 
            
  | 16:00 | 
  <elukey> | 
  stopping Hadoop daemons and shutting down analytics[1032-1033,1040].eqiad.wmnet - T132256 | 
  [production] | 
            
  | 15:20 | 
  <moritzm> | 
  installing rpcbind/libtirpc security updates on ms1001 | 
  [production] | 
            
  | 15:15 | 
  <moritzm> | 
  uploaded kubernetes 1.5.5-1+wmf1 to stretch-wikimedia/experimental | 
  [production] | 
            
  | 15:02 | 
  <urandom> | 
  starting instances restbase2005 | 
  [production] | 
            
  | 14:55 | 
  <moritzm> | 
  repooled mw1264 after hardware error has been fixed (and scap pull) | 
  [production] | 
            
  | 14:45 | 
  <hashar> | 
  European SWAT completed | 
  [production] | 
            
  | 14:39 | 
  <bblack> | 
  varnish: varnishadm runtime set default_ttl=86400 for text+upload fe+be layers via cumin, to match deployed start-time changes in https://gerrit.wikimedia.org/r/#/c/352826/ | 
  [production] |