| 
      
        2018-08-01
      
      ยง
     | 
  
    
  | 13:28 | 
  <marostegui> | 
  Deploy schema change on db1083 T144010 T51190 T199368 | 
  [production] | 
            
  | 13:28 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 55s) | 
  [production] | 
            
  | 13:02 | 
  <volans> | 
  restarting ircecho on einsteinium to unblock icinga IRC bot | 
  [production] | 
            
  | 12:59 | 
  <moritzm> | 
  installing tomcat7 security updates on jessie | 
  [production] | 
            
  | 12:35 | 
  <marostegui> | 
  Deploy schema change on db1105:3311 T144010 T51190 T199368 | 
  [production] | 
            
  | 12:35 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 (duration: 00m 55s) | 
  [production] | 
            
  | 12:30 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 (duration: 00m 55s) | 
  [production] | 
            
  | 12:16 | 
  <moritzm> | 
  installing fuse security updates | 
  [production] | 
            
  | 12:16 | 
  <marostegui> | 
  Deploy schema change on db1099:3311 T144010 T51190 T199368 | 
  [production] | 
            
  | 12:13 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 (duration: 00m 56s) | 
  [production] | 
            
  | 11:55 | 
  <zeljkof> | 
  EU SWAT finished | 
  [production] | 
            
  | 11:54 | 
  <zfilipin@deploy1001> | 
  Synchronized wmf-config/: SWAT: [[gerrit:449437|CleanupParent for draftquality model when PageTriage is used (T199357)]] (duration: 00m 56s) | 
  [production] | 
            
  | 11:45 | 
  <zfilipin@deploy1001> | 
  Synchronized php-1.32.0-wmf.15/skins/MinervaNeue/: SWAT: [[gerrit:449647|Restore page issues (T200867)]] (duration: 00m 57s) | 
  [production] | 
            
  | 11:30 | 
  <zfilipin@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:448553|Enable SandboxLink on the isiZulu Wikipedia (T200522)]] (duration: 00m 56s) | 
  [production] | 
            
  | 11:20 | 
  <zfilipin@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:449663|Fix config for $wgLocalisationUpdateRepositories (T148965)]] (duration: 00m 57s) | 
  [production] | 
            
  | 11:17 | 
  <moritzm> | 
  installing openjpeg2 security updates on jessie | 
  [production] | 
            
  | 10:44 | 
  <ema> | 
  reboot cp1045 for kernel update | 
  [production] | 
            
  | 10:40 | 
  <marostegui> | 
  Reload haproxy on dbproxy1008 and dbproxy1003 | 
  [production] | 
            
  | 09:50 | 
  <moritzm> | 
  installing wireshark security updates | 
  [production] | 
            
  | 09:50 | 
  <marostegui> | 
  Deploy schema change on db2048 (s1 codfw master) with replication, this will generate lag on codfw s1 T144010 T51190 T199368 | 
  [production] | 
            
  | 09:46 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s) | 
  [production] | 
            
  | 09:41 | 
  <ema> | 
  cp3044 (upload) repooled after upgrade to stretch T200445 | 
  [production] | 
            
  | 09:39 | 
  <marostegui> | 
  Run community_metrics.sh on phab1001 | 
  [production] | 
            
  | 09:36 | 
  <moritzm> | 
  installing clamav security updates | 
  [production] | 
            
  | 09:19 | 
  <marostegui> | 
  Deploy schema change on db1092 T144010 T51190 T199368 | 
  [production] | 
            
  | 09:18 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 55s) | 
  [production] | 
            
  | 09:14 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 56s) | 
  [production] | 
            
  | 08:38 | 
  <elukey> | 
  restart hadoop-yarn-nodemanager on analytics10[31-77] to apply the new memory settings | 
  [production] | 
            
  | 08:37 | 
  <marostegui> | 
  Deploy schema change on db1109:3318 T144010 T51190 T199368 | 
  [production] | 
            
  | 08:37 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s) | 
  [production] | 
            
  | 08:22 | 
  <Amir1> | 
  start of  ladsgroup@mwmaint1001:~$ foreachwikiindblist s1 populateChangeTagDef.php --sleep 2 (T193873) | 
  [production] | 
            
  | 08:21 | 
  <Amir1> | 
  start of  ladsgroup@mwmaint1001:~$ foreachwikiindblist s8 populateChangeTagDef.php --sleep 3 (T193873) | 
  [production] | 
            
  | 08:17 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 56s) | 
  [production] | 
            
  | 08:16 | 
  <marostegui> | 
  Stop MySQL on db2078 | 
  [production] | 
            
  | 07:59 | 
  <elukey> | 
  restart hadoop-yarn-nodemanager on analytics10[28-30] to test new memory settings | 
  [production] | 
            
  | 07:12 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 55s) | 
  [production] | 
            
  | 07:11 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 56s) | 
  [production] | 
            
  | 07:09 | 
  <marostegui> | 
  Remove db1052 from tendril - T199861 | 
  [production] | 
            
  | 06:59 | 
  <elukey> | 
  restart eventlogging on eventlog1002 to pick up new logging settings | 
  [production] | 
            
  | 06:58 | 
  <elukey@deploy1001> | 
  Finished deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/ (duration: 00m 07s) | 
  [production] | 
            
  | 06:58 | 
  <elukey@deploy1001> | 
  Started deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/ | 
  [production] | 
            
  | 06:47 | 
  <marostegui> | 
  Deploy schema change on db1087 with replication, this will generate lag on labs:s8 T144010 T51190 T199368 | 
  [production] | 
            
  | 06:47 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 55s) | 
  [production] | 
            
  | 06:41 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Increase traffic db1089 (duration: 00m 55s) | 
  [production] | 
            
  | 06:39 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 55s) | 
  [production] | 
            
  | 06:22 | 
  <brion> | 
  restarting transcode batch job (errors believed caused by hhvm config restarts) | 
  [production] | 
            
  | 06:11 | 
  <brion> | 
  stopping requeueTranscodes.php for now | 
  [production] | 
            
  | 06:11 | 
  <brion> | 
  seeing spike in errors on video scalers | 
  [production] | 
            
  | 05:14 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Slowly repool db1089 (duration: 00m 57s) | 
  [production] | 
            
  | 05:02 | 
  <marostegui> | 
  Stop MySQL on db1089 for a binlog format change (and also upgrade kernel) | 
  [production] |