| 2018-06-27
      
      § | 
    
  | 07:38 | <vgutierrez@puppetmaster1001> | conftool action : set/pooled=no; selector: name=dns4001.wikimedia.org | [production] | 
            
  | 07:37 | <vgutierrez> | Depool dns4001 for server restart - T198215 | [production] | 
            
  | 05:08 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 for alter table (duration: 01m 06s) | [production] | 
            
  | 05:08 | <marostegui> | Deploy schema change on db1090:3312 T191316 T192926 T89737 T195193 | [production] | 
            
  | 04:47 | <mholloway-shell@deploy1001> | Finished deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba (duration: 05m 42s) | [production] | 
            
  | 04:42 | <mholloway-shell@deploy1001> | Started deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba | [production] | 
            
  | 03:04 | <l10nupdate@deploy1001> | ResourceLoader cache refresh completed at Wed Jun 27 03:04:07 UTC 2018 (duration 10m 32s) | [production] | 
            
  | 02:53 | <l10nupdate@deploy1001> | scap sync-l10n completed (1.32.0-wmf.10) (duration: 15m 53s) | [production] | 
            
  | 02:20 | <l10nupdate@deploy1001> | scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 33s) | [production] | 
            
  
    | 2018-06-26
      
      § | 
    
  | 21:26 | <dduvall@deploy1001> | rebuilt and synchronized wikiversions files: Group0 to 1.32.0-wmf.10 | [production] | 
            
  | 21:18 | <dduvall@deploy1001> | Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 22m 50s) | [production] | 
            
  | 20:55 | <dduvall@deploy1001> | Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache | [production] | 
            
  | 20:47 | <marxarelli> | removing orphaned /srv/mediawiki/php-1.31.0-wmf.28 dir on deploy1001 | [production] | 
            
  | 20:38 | <volans> | regenerated debian installer for jessie 8.11 point release (released 3 days ago) | [production] | 
            
  | 20:34 | <marxarelli> | removed /srv/mediawiki/php-1.32.0-wmf.4 on mwdebug2002 to free disk space | [production] | 
            
  | 20:22 | <marxarelli> | manually cleaning up on mwdebug2002 due to full disk and scap failures | [production] | 
            
  | 20:21 | <dduvall@deploy1001> | Pruned MediaWiki: 1.32.0-wmf.4 [keeping static files] (duration: 03m 21s) | [production] | 
            
  | 19:43 | <marxarelli> | cdb-rebuild failed on mwdebug2002 due to lack of disk space | [production] | 
            
  | 19:41 | <dduvall@deploy1001> | Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 39m 11s) | [production] | 
            
  | 19:02 | <dduvall@deploy1001> | Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache | [production] | 
            
  | 18:39 | <marxarelli> | setting $wmgUseMwEmbedSupport = false in php-1.32.0-wmf.10/LocalSettings.php to extension registry exception (see T197918 and T191056) | [production] | 
            
  | 18:14 | <marxarelli> | scap sync for testwiki is currently failing on l10update due to MwEmbedSupport's deprecation but no change yet to mediawiki/extensions to remove the extension. blocking train on T197918 | [production] | 
            
  | 18:01 | <dduvall@deploy1001> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2905568126" --threads=30 --lang en  --quiet' returned non-zero exit status 255 (duration: 02m 23s) | [production] | 
            
  | 17:59 | <dduvall@deploy1001> | Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache | [production] | 
            
  | 17:16 | <marxarelli> | starting branch cut for 1.32-wmf.10 | [production] | 
            
  | 16:21 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 57s) | [production] | 
            
  | 14:43 | <elukey> | rm syslog.1.gz puppet.log.1.gz on tegment to fix cronspam | [production] | 
            
  | 14:19 | <ottomata> | Re-enable job and change-prop Kafka topic mirroring from main-eqiad -> main-codfw | [production] | 
            
  | 13:37 | <hashar> | European swat completed | [production] | 
            
  | 13:36 | <hashar@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Enable zh-my variant on zhwiki - T193983 (duration: 00m 57s) | [production] | 
            
  | 13:31 | <hashar@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Use uploaded HD logos for ruwikiquote - T197508 (duration: 00m 57s) | [production] | 
            
  | 13:29 | <hashar> | purgeList.php  for the various ruwikiquote logos | [production] | 
            
  | 13:27 | <hashar@deploy1001> | Synchronized static/images/project-logos: Change logo resources for ruwikiquote - T197508 (duration: 00m 56s) | [production] | 
            
  | 13:25 | <hashar@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on ruwikiquote - T197526 (duration: 00m 57s) | [production] | 
            
  | 13:22 | <dcausse> | elastic@eqiad: forcemerge (only_expunge_deletes=true) on wikidatawiki_content | [production] | 
            
  | 13:21 | <dcausse> | elastic@eqiad deleting stale index viwiki_general_1525301649 | [production] | 
            
  | 13:19 | <hashar@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for ruwiktionary - T197565 (duration: 00m 57s) | [production] | 
            
  | 13:06 | <hashar@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Add some namespace aliases to ruwikiquote - T197058 (duration: 00m 56s) | [production] | 
            
  | 12:54 | <chasemp> | labcontrol1003 aptitude install ldap-utils python-ldappool | [production] | 
            
  | 12:15 | <marostegui> | Deploy schema change on db1105:3312 T191316 T192926 T89737 T195193 | [production] | 
            
  | 12:14 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 57s) | [production] | 
            
  | 12:07 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 57s) | [production] | 
            
  | 11:09 | <godog> | roll-upgrade thumbor in codfw/eqiad | [production] | 
            
  | 11:08 | <godog> | upload thumbor 2.0-2 to apt.wikimedia.org | [production] | 
            
  | 10:40 | <godog> | upgrade thumbor to 2.0-1 on thumbor1001 | [production] | 
            
  | 10:10 | <godog> | test upgrade thumbor 2.0 on thumbor2001 | [production] | 
            
  | 10:10 | <akosiaris> | proceed with apertium-apy upgrade on all scb hosts. T194342 | [production] | 
            
  | 10:01 | <akosiaris> | downgrade erroneously upgraded apertium-fra-cat on scb1001, scb1002 from 1.3.0~r84327-1+wmf1 to 1.2.0~r78602-1+wmf2 T194342 | [production] | 
            
  | 09:40 | <oblivian@puppetmaster1001> | conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw | [production] | 
            
  | 09:39 | <akosiaris> | re-enable icinga-wm that I 've previously silenced | [production] |