| 
      
        2018-07-20
      
      §
     | 
  
    
  | 08:01 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 55s) | 
  [production] | 
            
  | 07:46 | 
  <marostegui> | 
  Stop replicaiton on s8 codfw master - T200061 | 
  [production] | 
            
  | 07:43 | 
  <jynus> | 
  stopping db1095 mariadb and cloning it to db1102 | 
  [production] | 
            
  | 07:14 | 
  <jynus@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1099, es1019 fully (duration: 00m 54s) | 
  [production] | 
            
  | 07:11 | 
  <marostegui> | 
  Stop replication in sync on db1074 and db1125:3312 | 
  [production] | 
            
  | 06:13 | 
  <marostegui> | 
  Deploy schema change on db1074 with replication, this will generate lag on labsdb:s2 T144010 T51190 T199368 | 
  [production] | 
            
  | 06:12 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 53s) | 
  [production] | 
            
  | 06:07 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1076 after alter table (duration: 00m 54s) | 
  [production] | 
            
  | 05:49 | 
  <marostegui> | 
  Deploy schema change on db1076 T144010 T51190 T199368 | 
  [production] | 
            
  | 05:44 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 00m 56s) | 
  [production] | 
            
  | 05:08 | 
  <marostegui> | 
  Start to remove some unused files on db1067 - T200039 | 
  [production] | 
            
  | 01:38 | 
  <krinkle@deploy1001> | 
  Synchronized php-1.32.0-wmf.13/extensions/CentralAuth/includes/CentralAuthUser.php: T170971 (duration: 00m 55s) | 
  [production] | 
            
  | 00:31 | 
  <krinkle@deploy1001> | 
  Synchronized php-1.32.0-wmf.13/includes/filerepo/: I58706b5610 and I40f6ad2a3d - T200026 (duration: 00m 56s) | 
  [production] | 
            
  | 00:30 | 
  <ejegg> | 
  updated CiviCRM from 22a8c47668 to d76e3997e6 | 
  [production] | 
            
  | 00:12 | 
  <bblack> | 
  disabled puppet temporary on cp* and kafka-jumbo* for ipsec unconfiguration | 
  [production] | 
            
  
    | 
      
        2018-07-19
      
      §
     | 
  
    
  | 23:43 | 
  <ejegg> | 
  updated CiviCRM from 08680be68e to 22a8c47668 | 
  [production] | 
            
  | 23:31 | 
  <ejegg> | 
  updated CiviCRM from 2bb0d0f9d0 to 08680be68e | 
  [production] | 
            
  | 23:11 | 
  <thcipriani@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:446982|Revert "Enable Special:Block Feedback Request"]] (duration: 00m 55s) | 
  [production] | 
            
  | 22:51 | 
  <XioNoX> | 
  disable ospf on eqdfw-knams link (GTT) | 
  [production] | 
            
  | 22:18 | 
  <XioNoX> | 
  enable ospf on eqdfw-knams link (GTT) | 
  [production] | 
            
  | 21:52 | 
  <hashar> | 
  CI Jenkins has been upgraded successfully! | 
  [production] | 
            
  | 21:21 | 
  <hashar> | 
  restarting CI on contint1001 and actually upgrading it | 
  [production] | 
            
  | 21:19 | 
  <hashar> | 
  starting CI jenkins on contint1001 | 
  [production] | 
            
  | 21:16 | 
  <hashar> | 
  stopping the CI jenkins on contint1001 | 
  [production] | 
            
  | 20:20 | 
  <jynus@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1099 with low load (duration: 00m 54s) | 
  [production] | 
            
  | 19:25 | 
  <ariel@deploy1001> | 
  Finished deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps (duration: 00m 03s) | 
  [production] | 
            
  | 19:25 | 
  <ariel@deploy1001> | 
  Started deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps | 
  [production] | 
            
  | 19:21 | 
  <ejegg> | 
  updated CiviCRM from 6f311f48d9 to 2bb0d0f9d0 | 
  [production] | 
            
  | 19:18 | 
  <XioNoX> | 
  setting mtu 9192 to all codfw interface ranges | 
  [production] | 
            
  | 19:04 | 
  <elukey> | 
  roll restart eventstreams on scb2* hosts to prevent OOM issues over the EU night - T199813 | 
  [production] | 
            
  | 19:00 | 
  <Amir1> | 
  Morning SWAT is done | 
  [production] | 
            
  | 18:57 | 
  <ladsgroup@deploy1001> | 
  Synchronized wmf-config/abusefilter.php: SWAT: [[gerrit:446872|Add abusefilter-modify-restricted right to sysops on itwiktionary (T199783)]] (duration: 00m 53s) | 
  [production] | 
            
  | 18:50 | 
  <ladsgroup@deploy1001> | 
  Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: [[gerrit:446813|Report SPARQL errors as TODO, not VIOLATION (T199788)]] (duration: 00m 56s) | 
  [production] | 
            
  | 18:27 | 
  <ladsgroup@deploy1001> | 
  Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: [[gerrit:446813|Report SPARQL errors as TODO, not VIOLATION (T199788)]] (duration: 00m 56s) | 
  [production] | 
            
  | 18:17 | 
  <Amir1> | 
  start of ladsgroup@mwmaint1001:~$ foreachwikiindblist all populateChangeTagDef.php (T193873) It will take a couple of days | 
  [production] | 
            
  | 18:14 | 
  <ladsgroup@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:446804|Set to write the new change tag backend everywhere, enable reading in frwiki (T194165)]] (duration: 00m 55s) | 
  [production] | 
            
  | 16:28 | 
  <jynus> | 
  stop, clone away and upgrade db1099 two mysql instances | 
  [production] | 
            
  | 16:16 | 
  <jynus@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Fully depool db1099, including from s8 (duration: 00m 55s) | 
  [production] | 
            
  | 15:59 | 
  <hoo> | 
  Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds | 
  [production] | 
            
  | 15:53 | 
  <hashar> | 
  CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes | https://gerrit.wikimedia.org/r/#/c/446863 | T198348 | 
  [production] | 
            
  | 15:50 | 
  <jynus> | 
  droping undocumented grants on labsdb1005 using replication T199186 | 
  [production] | 
            
  | 15:21 | 
  <volans> | 
  reimaging mw2226 to test py3 reimage with new cumin, conftool and apache test from neodymium - T187773 | 
  [production] | 
            
  | 15:12 | 
  <volans@sarin> | 
  conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet | 
  [production] | 
            
  | 15:11 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 54s) | 
  [production] | 
            
  | 15:00 | 
  <jynus@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 54s) | 
  [production] | 
            
  | 14:59 | 
  <moritzm> | 
  rebooting multatuli for kernel test | 
  [production] | 
            
  | 14:30 | 
  <marostegui> | 
  Deploy schema change on db1105:3312 T144010 T51190 T199368 | 
  [production] | 
            
  | 14:29 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 54s) | 
  [production] | 
            
  | 14:22 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 54s) | 
  [production] | 
            
  | 14:17 | 
  <elukey> | 
  roll restart kafka on kafka-jumbo* and kafka main-codfw (kafka2*) to pick up new Xmx/Xms settings (1g -> 2g) | 
  [production] |