| 2018-11-16
      
      § | 
    
  | 06:36 | <marostegui@deploy1001> | Synchronized wmf-config/db-codfw.php: Pool pc2009 in pc3 - T208383 (duration: 00m 56s) | [production] | 
            
  | 06:28 | <marostegui> | Set sync_binlog=0 and trx_commit=2 on dbstore2002:3313 to let it catch up | [production] | 
            
  | 05:47 | <vgutierrez> | uploaded certcentral 0.7 to apt.wikimedia.org (stretch) - T208967 T209475 | [production] | 
            
  | 00:55 | <mutante> | some users reported missing files in home dirs on mwmaint1002, reversed rsyncd/ferm setup and rsynced /home from mwmaint2001 to /root on mwmaint1002, restored individually where requested, rsync is not fully automatic but puppetized with rsync::quickdatacopy | [production] | 
            
  | 00:35 | <sbisson@deploy1001> | Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/PageViews.php: SWAT: [[gerrit:473861|Exclude users where getRegistration() returns null]] (duration: 00m 47s) | [production] | 
            
  | 00:26 | <eileen> | civicrm revision changed from 71755d021b to 6b279509f8, config revision is 684ec9b7c0 (lybunt report) | [production] | 
            
  | 00:22 | <sbisson@deploy1001> | Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments: SWAT: [[gerrit:473843]] [[gerrit:473844]] [[gerrit:473845]] (duration: 00m 49s) | [production] | 
            
  | 00:21 | <sbisson@deploy1001> | sync aborted: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: [[gerrit:473843]] [[gerrit:473844]] [[gerrit:473845]] (duration: 06m 16s) | [production] | 
            
  | 00:15 | <sbisson@deploy1001> | Started scap: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: [[gerrit:473843]] [[gerrit:473844]] [[gerrit:473845]] | [production] | 
            
  
    | 2018-11-15
      
      § | 
    
  | 21:58 | <mutante> | mwmaint1002 - restoring entire /home of mwmaint1001 from Bacula (job queued and to tmp dir, not directly into /home) | [production] | 
            
  | 21:06 | <hashar> | Deleting Nodepool instances on contintcloud T209361 | [production] | 
            
  | 21:05 | <hashar> | Stopped nodepool on labnodepool1001.eqiad.wmnet . Service is no more used. T209361 T209642 | [production] | 
            
  | 20:22 | <mholloway-shell@deploy1001> | Finished deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 03m 55s) | [production] | 
            
  | 20:18 | <mholloway-shell@deploy1001> | Started deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) | [production] | 
            
  | 20:17 | <mutante> | re-added Chase to pwstore, signed .users file, re-encrypted all pwstore files, git pushed | [production] | 
            
  | 20:15 | <mholloway-shell@deploy1001> | Finished deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 04m 26s) | [production] | 
            
  | 20:11 | <mholloway-shell@deploy1001> | Started deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) | [production] | 
            
  | 20:04 | <urandom> | dropping disused keyspaces -- T208616 | [production] | 
            
  | 20:04 | <catrope@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Enable data collection for UnderstandingFirstDay on cswiki and kowiki (duration: 00m 53s) | [production] | 
            
  | 19:51 | <catrope@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Configure sensitive namespaces for EditorJourney schema (T207307) (duration: 00m 53s) | [production] | 
            
  | 19:49 | <catrope@deploy1001> | Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: cherry-picks for T208773 (duration: 00m 54s) | [production] | 
            
  | 18:55 | <jforrester@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Hot-deploy I26f2dc2e: Don't over-ride default Wikibase string limits (duration: 00m 53s) | [production] | 
            
  | 18:50 | <jforrester@deploy1001> | Synchronized wmf-config/Wikibase.php: Hot-deploy Ib10de2e3: Don't set Wikibase string limits when null (duration: 00m 55s) | [production] | 
            
  | 18:31 | <ladsgroup@deploy1001> | Finished deploy [ores/deploy@dba11e9]: Another small update (duration: 13m 42s) | [production] | 
            
  | 18:18 | <ladsgroup@deploy1001> | Started deploy [ores/deploy@dba11e9]: Another small update | [production] | 
            
  | 18:17 | <ladsgroup@deploy1001> | Finished deploy [ores/deploy@51cdf6b]: T208623 (duration: 14m 41s) | [production] | 
            
  | 18:03 | <ladsgroup@deploy1001> | Started deploy [ores/deploy@51cdf6b]: T208623 | [production] | 
            
  | 17:27 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 7 wikis in group 2 for T209373. This may cause lag in codfw. | [production] | 
            
  | 17:27 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 6 wikis in group 2 for T209373. This may cause lag in codfw. | [production] | 
            
  | 17:27 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 5 wikis in group 2 for T209373. This may cause lag in codfw. | [production] | 
            
  | 17:27 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 3 wikis in group 2 for T209373. This may cause lag in codfw. | [production] | 
            
  | 17:27 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 2 wikis in group 2 for T209373. This may cause lag in codfw. | [production] | 
            
  | 17:27 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 1 wikis in group 2 for T209373. This may cause lag in codfw. | [production] | 
            
  | 17:20 | <gehel> | upgrade prometheus-blazegraph-exporter on all wdqs nodes - T206123 | [production] | 
            
  | 16:41 | <bstorm_> | rebooted labsdb1007 for upgrades | [production] | 
            
  | 16:37 | <andrewbogott> | rebuilding labvirt1015 and cloudvirt1015 | [production] | 
            
  | 14:21 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on wikitech for T209373. This may cause lag in codfw. | [production] | 
            
  | 14:21 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 8 wikis in group 1 for T209373. This may cause lag in codfw. | [production] | 
            
  | 14:21 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 7 wikis in group 1 for T209373. This may cause lag in codfw. | [production] | 
            
  | 14:21 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 5 wikis in group 1 for T209373. This may cause lag in codfw. | [production] | 
            
  | 14:21 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 4 wikis in group 1 for T209373. This may cause lag in codfw. | [production] | 
            
  | 14:21 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 3 wikis in group 1 for T209373. This may cause lag in codfw. | [production] | 
            
  | 14:20 | <anomie@mwmaint1002> | Running refreshExternallinksIndex.php on section 2 wikis in group 1 for T209373. This may cause lag in codfw. | [production] | 
            
  | 14:07 | <gehel> | plugin and JVM upgrade on elasticsearch / cirrus / eqiad completed - T209293 | [production] | 
            
  | 13:59 | <hashar@deploy1001> | rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.4 | [production] | 
            
  | 13:55 | <XioNoX> | push firewall policies to pfw3-eqiad - T209421 | [production] | 
            
  | 13:50 | <banyek> | Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T85757 | [production] | 
            
  | 13:48 | <moritzm> | installing qemu security updates (which also backport support for SSBD passthrough) on ganeti clusters | [production] | 
            
  | 13:48 | <moritzm> | installing qemu security updates (which also backport support for SSBD passthrough) | [production] | 
            
  | 13:12 | <mobrovac@deploy1001> | Finished deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477 (duration: 19m 56s) | [production] |