| 2018-07-10
      
      § | 
    
  | 10:07 | <ppchelko@deploy1001> | Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header | [production] | 
            
  | 10:07 | <moritzm> | restarting thumbor on thumbor1001 to pick up exiv2 security updates | [production] | 
            
  | 10:03 | <elukey> | restart analytics100[1,2]'s hadoop resource managers, some I/O socket errors after the ip6 interface change | [production] | 
            
  | 09:34 | <elukey> | forced umount of /mnt/hdfs on stat1004, several processes hang for it (causing load) and transport not connected | [production] | 
            
  | 09:25 | <godog> | swift eqiad-prod add ms-be1036 back gradually - T196873 | [production] | 
            
  | 09:05 | <moritzm> | installing libsoup security updates on jessie/stretch | [production] | 
            
  | 08:38 | <moritzm> | installing libjpeg-turbo security updates on trusty | [production] | 
            
  | 08:37 | <godog> | drain and restart cassandra-a on restbase2001 to test a restart | [production] | 
            
  | 08:34 | <elukey> | rolling restart of AQS to apply the new config | [production] | 
            
  | 08:09 | <godog> | disable puppet on hosts running cassandra before merging 444247 and 443114 | [production] | 
            
  | 08:08 | <moritzm> | installing openslp security updates on trusty | [production] | 
            
  | 07:58 | <moritzm> | installing ntp security updates on trusty (may trigger some Icinga warnings about clocks, these recover after a while) | [production] | 
            
  | 07:33 | <twentyafterfour> | deploying fix for phabricator userpage having hard-coded my username | [production] | 
            
  | 06:53 | <twentyafterfour> | deployed rPHEX03173dd0097451f60faa3a1705abee58a9fe4c5f | [production] | 
            
  | 06:32 | <marostegui> | Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw - T187521 | [production] | 
            
  | 06:23 | <marostegui> | Deploy schema change on codfw s7 master (db2040) with replication, this will generate lag on s7 codfw T146591 T197891 T196379 | [production] | 
            
  | 06:17 | <marostegui> | Deploy schema change on s8 primary master (db1071) T146591 T197891 T196379 | [production] | 
            
  | 06:06 | <marostegui> | Deploy schema change on dbstore1002:s8 T146591 T197891 T196379 | [production] | 
            
  | 06:02 | <marostegui> | Deploy schema change on codfw s8 master (db2045) with replication, this will generate lag on s8 codfw T146591 T197891 T196379 | [production] | 
            
  | 05:40 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 50s) | [production] | 
            
  | 05:39 | <marostegui> | Deploy schema change on s4 primary master (db1068) T146591 T197891 T196379 | [production] | 
            
  | 05:36 | <marostegui> | Deploy schema change on db1091 T146591 T197891 T196379 | [production] | 
            
  | 05:36 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 50s) | [production] | 
            
  | 05:32 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 49s) | [production] | 
            
  | 05:27 | <marostegui> | Deploy schema change on db1081 T146591 T197891 T196379 | [production] | 
            
  | 05:27 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 50s) | [production] | 
            
  | 05:21 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s) | [production] | 
            
  | 05:21 | <marostegui> | Deploy schema change on db1121 with replication, this will generate lag on s4 labs hosts T146591 T197891 T196379 | [production] | 
            
  | 05:17 | <marostegui> | Optimize frwiki.wbc_entity_usage on s6 eqiad hosts T187521 | [production] | 
            
  | 05:16 | <marostegui> | Deploy schema change on db1103:3314 T146591 T197891 T196379 | [production] | 
            
  | 05:16 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s) | [production] | 
            
  | 05:10 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 50s) | [production] | 
            
  | 05:03 | <marostegui> | Deploy schema change on db1097:3314 T146591 T197891 T196379 | [production] | 
            
  | 05:03 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 51s) | [production] | 
            
  | 04:59 | <marostegui> | Optimize frwiki.wbc_entity_usage on s6 codfw, this will generate lag on s6 codfw - T187521 | [production] | 
            
  | 04:57 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s) | [production] | 
            
  | 04:48 | <marostegui> | Deploy schema change on db1084 T146591 T197891 T196379 | [production] | 
            
  | 04:48 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 52s) | [production] | 
            
  | 04:36 | <marostegui> | Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1066 (s2 primary master) - T187521 | [production] | 
            
  | 04:28 | <marostegui> | Deploy schema change on s1 primary master (db1052) T146591 T197891 T196379 | [production] | 
            
  | 02:44 | <l10nupdate@deploy1001> | ResourceLoader cache refresh completed at Tue Jul 10 02:44:29 UTC 2018 (duration 10m 18s) | [production] | 
            
  | 00:21 | <twentyafterfour> | deploying https://phabricator.wikimedia.org/rPHABc6f75c918afa1cc59472c5fe226539e093f6c3ef | [production] | 
            
  
    | 2018-07-09
      
      § | 
    
  | 23:58 | <ejegg> | updated payments-wiki from ed40f33c44 to 383f667171 | [production] | 
            
  | 23:58 | <jforrester@deploy1001> | Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop trying to set wgLicenseURL T154069 (duration: 00m 50s) | [production] | 
            
  | 23:55 | <jforrester@deploy1001> | Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: No need for officewiki-specific upload for MP3s any more I42ffbcd76f6 (duration: 00m 50s) | [production] | 
            
  | 23:53 | <jforrester@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgTmhEnableMp3Uploads Idbf1201813 (duration: 00m 50s) | [production] | 
            
  | 23:52 | <jforrester@deploy1001> | Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgTmhEnableMp3Uploads I6479dbacd4 (duration: 00m 50s) | [production] | 
            
  | 23:48 | <jforrester@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion I3b8cccc00 (duration: 00m 50s) | [production] | 
            
  | 23:47 | <jforrester@deploy1001> | Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion If2ee24ae (duration: 00m 50s) | [production] | 
            
  | 23:43 | <jforrester@deploy1001> | Synchronized wmf-config/CommonSettings-labs.php: SWAT Cleanup: Remove Beta Cluster use of wikieditor-preview preference Ib16e6e19b1 (duration: 00m 50s) | [production] |