| 
      
        2017-05-16
      
      ยง
     | 
  
    
  | 20:10 | 
  <mobrovac@tin> | 
  Started restart [restbase/deploy@d98af6f] (dev-cluster): Apply the revision range delition algorithm - T164865 | 
  [production] | 
            
  | 18:49 | 
  <jynus> | 
  rolled back to HEAD~2 on tin to leave things the way I found them | 
  [production] | 
            
  | 18:41 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1055 after reimage (duration: 00m 39s) | 
  [production] | 
            
  | 18:26 | 
  <bblack> | 
  cp1074: run-no-puppet varnish-backend-restart (has high mailbox lag, causing small 503 spikes) | 
  [production] | 
            
  | 17:23 | 
  <cmjohnson1> | 
  swapping optics asw-c-eqiad xe-8/0/38 T165008 | 
  [production] | 
            
  | 17:05 | 
  <moritzm> | 
  upgrading mw2017/mw2099 to Linux 4.9 and HHVM 3.18 | 
  [production] | 
            
  | 16:40 | 
  <moritzm> | 
  upgrading mw2190-mw2199 to Linux 4.9 and HHVM 3.18 | 
  [production] | 
            
  | 16:22 | 
  <robh@puppetmaster1001> | 
  conftool action : set/pooled=no; selector: name=mw2098.codfw.wmnet | 
  [production] | 
            
  | 15:48 | 
  <jynus> | 
  restarting and upgrading db1095 | 
  [production] | 
            
  | 15:01 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Remove old comment (duration: 00m 39s) | 
  [production] | 
            
  | 14:53 | 
  <marostegui> | 
  Deploy alter table on s2 (revision table) db2041 - https://phabricator.wikimedia.org/T162611 | 
  [production] | 
            
  | 14:53 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Repool db2056, depool db2041 - T162611 (duration: 00m 41s) | 
  [production] | 
            
  | 14:50 | 
  <mobrovac@tin> | 
  Started restart [restbase/deploy@d98af6f]: Apply new puppet role/profile paradigm | 
  [production] | 
            
  | 14:36 | 
  <kartik@tin> | 
  Finished deploy [cxserver/deploy@6118dda]: Update cxserver to 740641f (duration: 02m 21s) | 
  [production] | 
            
  | 14:34 | 
  <kartik@tin> | 
  Started deploy [cxserver/deploy@6118dda]: Update cxserver to 740641f | 
  [production] | 
            
  | 14:27 | 
  <moritzm> | 
  upgrading mw2180-mw2189 to Linux 4.9 and HHVM 3.18 | 
  [production] | 
            
  | 14:08 | 
  <jynus> | 
  rolling restart labsdb1009,10,11 for mariadb upgrade (and kernel upgrade) | 
  [production] | 
            
  | 14:06 | 
  <moritzm> | 
  rebooting restbase2007 for update to Linux 4.9 and to pick up openjdk security updates | 
  [production] | 
            
  | 13:53 | 
  <moritzm> | 
  upgrading mw2170-mw2179 to Linux 4.9 and HHVM 3.18 | 
  [production] | 
            
  | 13:48 | 
  <addshore> | 
  SWAT done | 
  [production] | 
            
  | 13:48 | 
  <addshore@tin> | 
  Synchronized php-1.30.0-wmf.1/extensions/QuickSurveys/extension.json: SWAT: [[gerrit:353978|Explicitly add mediawiki.cookie dependency]] (duration: 00m 39s) | 
  [production] | 
            
  | 13:40 | 
  <moritzm> | 
  rebooting restbase2006 for update to Linux 4.9 and to pick up openjdk security updates | 
  [production] | 
            
  | 13:39 | 
  <addshore@tin> | 
  Synchronized wmf-config/throttle.php: SWAT: [[gerrit:353921|Raise the account creation limit for www.enwp.org/WP:Meetup/Eugene/WikiAPA]] T165421 (duration: 00m 39s) | 
  [production] | 
            
  | 13:36 | 
  <addshore@tin> | 
  Synchronized wmf-config/: SWAT: [[gerrit:353965|#1]] T164502, [[gerrit:353973|#2]], [[gerrit:353969|#3]] (duration: 00m 41s) | 
  [production] | 
            
  | 13:19 | 
  <moritzm> | 
  upgrading mw2163-mw2169 to HHVM 3.18 | 
  [production] | 
            
  | 13:07 | 
  <moritzm> | 
  upgrading mw2110-mw2117 to HHVM 3.18 | 
  [production] | 
            
  | 12:55 | 
  <marostegui> | 
  Run pt-table-checksum on s7.centralauth - https://phabricator.wikimedia.org/T163190 | 
  [production] | 
            
  | 12:06 | 
  <marostegui> | 
  Deploy alter table on s2 (revision table) db2049 - https://phabricator.wikimedia.org/T162611 | 
  [production] | 
            
  | 12:05 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Depool db2049 - T162611 (duration: 00m 39s) | 
  [production] | 
            
  | 11:37 | 
  <moritzm> | 
  upgrading mw1190-mw1208 to Linux 4.9 and HHVM 3.18 | 
  [production] | 
            
  | 11:28 | 
  <jynus> | 
  stopping db1055 before reimage for backup | 
  [production] | 
            
  | 11:27 | 
  <Amir1> | 
  ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/CleanDuplicateScores.php --wiki=enwiki | 
  [production] | 
            
  | 11:25 | 
  <Amir1> | 
  cleaning up is completely done current number of rows: 9,261,264 T159753 | 
  [production] | 
            
  | 11:24 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1055 for reimage (duration: 00m 39s) | 
  [production] | 
            
  | 10:56 | 
  <moritzm> | 
  upgrading codfw app servers already using HHVM 3.18 to 3.18.2+wmf3 | 
  [production] | 
            
  | 10:49 | 
  <marostegui> | 
  Deploy schema change on testwikidatawiki.wb_terms on s3 codfw master - T165246 | 
  [production] | 
            
  | 10:36 | 
  <jynus> | 
  upgrading and restarting db2062's mariadb service | 
  [production] | 
            
  | 10:30 | 
  <moritzm> | 
  installing openjdk-7 security updates on trusty hosts | 
  [production] | 
            
  | 10:28 | 
  <addshore> | 
  T164407 addshore@terbium mwscriptwikiset extensions/Cognate/maintenance/populateCognatePages.php wiktionary.dblist --batch-size=1000 | 
  [production] | 
            
  | 10:27 | 
  <addshore> | 
  addshore@terbium mwscriptwikiset extensions/Cognate/maintenance/populateCognatePages.php wiktionary.dblist --batch-size=1000 | 
  [production] | 
            
  | 10:14 | 
  <moritzm> | 
  upgrading mw1185-mw1189 to Linux 4.9 and HHVM 3.18 | 
  [production] | 
            
  | 09:26 | 
  <moritzm> | 
  upgrading mw1189 / mw1293 from HHVM 3.18.2+wmf2 to 3.18.2+wmf3 | 
  [production] | 
            
  | 08:59 | 
  <moritzm> | 
  upgrading mw1170-mw1184 from HHVM 3.18.2+wmf2 to 3.18.2+wmf3 | 
  [production] | 
            
  | 08:45 | 
  <moritzm> | 
  upgrading git packages on tin/naos from local 2.11 backport to the version from jessie-backports | 
  [production] | 
            
  | 08:22 | 
  <moritzm> | 
  installing git security updates on trusty (jessie already fixed) | 
  [production] | 
            
  | 07:39 | 
  <godog> | 
  upload prometheus-mysqld-exporter 0.10.0 to jessie-wikimedia - T161296 | 
  [production] | 
            
  | 07:10 | 
  <moritzm> | 
  upgrading mw1261-mw1265 to HHVM 3.18.2+wmf3 | 
  [production] | 
            
  | 07:06 | 
  <Amir1_> | 
  start of cleaning up ores_classification table in enwiki last round (four hours) (T159753) | 
  [production] | 
            
  | 06:58 | 
  <moritzm> | 
  restarted hhvm on mw1165 (stuck in HPHP::Treadmill deadlock) | 
  [production] | 
            
  | 06:37 | 
  <marostegui> | 
  Stop replication at the same position on db1044 and db2018 - https://phabricator.wikimedia.org/T147166 https://phabricator.wikimedia.org/T130067 | 
  [production] |