| 
      
        2018-04-12
      
      §
     | 
  
    
  | 13:31 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool es1013 (duration: 01m 17s) | 
  [production] | 
            
  | 13:22 | 
  <gehel> | 
  i18n maps will not be available yet, this is only preliminary work | 
  [production] | 
            
  | 13:22 | 
  <gehel> | 
  deploying maps internationalization, including new keyspace and generating new tiles - T191655 | 
  [production] | 
            
  | 13:18 | 
  <zeljkof> | 
  EU SWAT finished | 
  [production] | 
            
  | 13:15 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:425588|Enable Page Previews for 10% enwiki anon users (T189906)]] (duration: 01m 18s) | 
  [production] | 
            
  | 13:01 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool es1012 with full weight (duration: 01m 17s) | 
  [production] | 
            
  | 12:51 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 01m 17s) | 
  [production] | 
            
  | 12:18 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1114 from API - T191996 (duration: 01m 17s) | 
  [production] | 
            
  | 12:14 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool es1012 with low weight (duration: 01m 19s) | 
  [production] | 
            
  | 12:13 | 
  <marostegui> | 
  Deploy schema change on s8 dbstore1002 - T187089 T185128 T153182 | 
  [production] | 
            
  | 11:59 | 
  <moritzm> | 
  pooling mw1279 for some brief test production traffic | 
  [production] | 
            
  | 09:58 | 
  <jynus> | 
  reimage es1012, take 2 | 
  [production] | 
            
  | 08:12 | 
  <marostegui> | 
  Drop table linkscc from s3 codfw primary master | 
  [production] | 
            
  | 08:11 | 
  <marostegui> | 
  Drop table linkscc from s1 | 
  [production] | 
            
  | 07:55 | 
  <marostegui> | 
  Drop table linkscc from s2 and s7 | 
  [production] | 
            
  | 07:50 | 
  <marostegui> | 
  Drop table linkscc from s4,s5 and s6 | 
  [production] | 
            
  | 07:41 | 
  <jynus> | 
  reimage es1012 | 
  [production] | 
            
  | 07:40 | 
  <moritzm> | 
  enabling production traffic for mw1265 | 
  [production] | 
            
  | 07:17 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table - T190780 (duration: 01m 16s) | 
  [production] | 
            
  | 07:11 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table - T190780 (duration: 01m 17s) | 
  [production] | 
            
  | 06:57 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1077 after alter table - T190780 (duration: 01m 17s) | 
  [production] | 
            
  | 06:53 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table - T190780 (duration: 01m 17s) | 
  [production] | 
            
  | 06:48 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1072 after alter table - T190780 (duration: 01m 16s) | 
  [production] | 
            
  | 06:42 | 
  <marostegui> | 
  Deploy schema change on db1072 (sanitarium master for s3)  - this will generate lag on s3 labsdb - T190780 | 
  [production] | 
            
  | 06:36 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1072 for alter table - T190780 (duration: 01m 18s) | 
  [production] | 
            
  | 06:27 | 
  <marostegui> | 
  Deploy schema change on s3 codfw master (db2043) - this will generate lag on s3 codfw -T190780 | 
  [production] | 
            
  | 06:24 | 
  <marostegui> | 
  Deploy schema change on s1 primary master (db1052) - T190780 | 
  [production] | 
            
  | 06:11 | 
  <marostegui> | 
  Deploy schema change on s7 primary master (db1062) - T190780 | 
  [production] | 
            
  | 06:08 | 
  <elukey> | 
  force kill of fuse_dfs (handling /mnt/hdfs) on stat1004, apparently causing a huge load | 
  [production] | 
            
  | 06:05 | 
  <elukey> | 
  force kill of fuse_dfs (handling /mnt/hdfs) on stat1005, apparently causing a huge load | 
  [production] | 
            
  | 05:52 | 
  <marostegui> | 
  Deploy schema change on s2 primary master (db1054) - T190780 | 
  [production] | 
            
  | 05:49 | 
  <marostegui> | 
  Deploy schema change on s8 primary master (db1071) - T190780 | 
  [production] | 
            
  | 05:45 | 
  <marostegui> | 
  Deploy schema change on s4 primary master (db1068) - T190780 | 
  [production] | 
            
  | 05:39 | 
  <marostegui> | 
  Deploy schema change on s6 primary master (db1061) - T190780 | 
  [production] | 
            
  | 05:34 | 
  <marostegui> | 
  Deploy schema change on s5 primary master (db1070) - T190780 | 
  [production] | 
            
  | 05:27 | 
  <marostegui> | 
  Deploy schema change on db1109 - T187089 T185128 T153182 | 
  [production] | 
            
  | 05:25 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 01m 17s) | 
  [production] | 
            
  | 05:15 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 01m 18s) | 
  [production] | 
            
  | 05:11 | 
  <marostegui> | 
  Reload haproxy on dbproxy1011 to repool labsdb1009 | 
  [production] | 
            
  | 02:38 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.31.0-wmf.28) (duration: 07m 20s) | 
  [production] | 
            
  | 01:34 | 
  <eileen> | 
  civicrm revision changed from 07bade75a2 to b3326dbf70, config revision is 853fcc9111 (deploy wmffraud report) | 
  [production] | 
            
  | 00:44 | 
  <twentyafterfour> | 
  The hotfix that I deployed for phabricator: https://phabricator.wikimedia.org/rPHEX7801b519442eea2bfd47a272ba36959b487ae7d7 | 
  [production] | 
            
  | 00:33 | 
  <twentyafterfour> | 
  phabricator: hotfixing DeadlineEditEngineSubtype.php | 
  [production] | 
            
  | 00:23 | 
  <twentyafterfour> | 
  phabricator is back | 
  [production] | 
            
  | 00:18 | 
  <twentyafterfour> | 
  phabricator will be offline for just a moment while I run the upgrade script. | 
  [production] | 
            
  | 00:15 | 
  <twentyafterfour> | 
  preparing to deploy phabricator rPHDEP/release/2018-04-12/1 https://phabricator.wikimedia.org/project/view/3335/ | 
  [production] | 
            
  | 00:09 | 
  <mutante> | 
  jerkins-bot tests all return -1 due to operations-mw-config-php55lint failing which says it can't clone on integration-slave-jessie-1003, which is out of disk space in /srv as reported by shinken. it's mostly all /srv/pbuilder | 
  [production] | 
            
  | 00:08 | 
  <twentyafterfour> | 
  phabricator update will begin shortly, running a bit behind due to a massive upstream merge which will have to wait until later date. | 
  [production] | 
            
  | 00:08 | 
  <maxsem@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/425723/ (duration: 01m 18s) | 
  [production] |