| 2016-08-24
      
      § | 
    
  | 13:39 | <ema> | upgrading cp4005 to Varnish 4 (T131502) | [production] | 
            
  | 13:27 | <zfilipin@tin> | Synchronized wmf-config/InitialiseSettings.php: Make default ORES threshold soft (higher threshold) T143738 (duration: 00m 59s) | [production] | 
            
  | 12:56 | <mobrovac> | changeprop deploying c5fd932 | [production] | 
            
  | 12:29 | <gehel> | remove old logstash indices that were not deleted after 31 days | [production] | 
            
  | 11:10 | <godog> | enable collection of mysqld metrics from prometheus2002 too | [production] | 
            
  | 10:49 | <moritzm> | start installing hhvm updates in eqiad | [production] | 
            
  | 10:13 | <gehel> | increase recovery throttling on elasticsearch codfw to reduce rolling restart time | [production] | 
            
  | 09:52 | <moritzm> | disabled creation of unprivileged user namespaces on trusty systems via sysctl kernel.unprivileged_userns_clone | [production] | 
            
  | 09:30 | <gehel> | starting rolling restart of elasticearch codfw for JVM and elasticsearch upgrade | [production] | 
            
  | 09:03 | <hashar> | gallium contint::firewall: Limited to production networks https://gerrit.wikimedia.org/r/301627  .  For monitoring do:  grep iptables-dropped /var/log/syslog | [production] | 
            
  | 08:39 | <ema> | dns: re-enabling codfw | [production] | 
            
  | 08:29 | <paravoid> | disabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), flapping since 07:31 UTC | [production] | 
            
  | 08:09 | <ema> | disabling codfw in dns | [production] | 
            
  | 03:07 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Wed Aug 24 03:07:38 UTC 2016 (duration 6m 55s) | [production] | 
            
  | 03:00 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.16) (duration: 18m 24s) | [production] | 
            
  | 02:25 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 16s) | [production] | 
            
  | 01:57 | <mutante> | temp stop dhcp service on install2001 - debug | [production] | 
            
  
    | 2016-08-23
      
      § | 
    
  | 23:45 | <dereckson@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable Related Articles on fr.wikinews (T143480) (duration: 00m 53s) | [production] | 
            
  | 23:28 | <mutante> | restarted grrrit-wm, apache and gerrit on lead | [production] | 
            
  | 23:23 | <mutante> | gerrit restarting for config changes 303355, 304977 | [production] | 
            
  | 22:04 | <bd808> | Updated tin:/srv/deployment/logstash/plugins to d18b1c6 (Add output plugin for Sentry) | [production] | 
            
  | 21:27 | <SMalyshev> | redeploying WDQS GUI to fix examples breakage | [production] | 
            
  | 21:27 | <dzahn@palladium> | conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,name=hydrogen.wikimedia.org | [production] | 
            
  | 21:05 | <mutante> | hydrogen - reinstall finished, re-added to salt, restarted ntpd | [production] | 
            
  | 20:22 | <mutante> | hydrogen - reinstalling one more time, wrong partitioning | [production] | 
            
  | 19:55 | <mutante> | re-signing new puppet certs for hydrogen, initial run, new salt key | [production] | 
            
  | 19:34 | <hashar> | 1.28.0-wmf.16 to group0  looks successful. | [production] | 
            
  | 19:19 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: fix compile & sync of wikiversions | [production] | 
            
  | 19:17 | <hashar@tin> | Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 50s) | [production] | 
            
  | 19:13 | <hashar@tin> | Synchronized wikiversions.json: Group0 to 1.28.0-wmf.16 T141551 (duration: 00m 48s) | [production] | 
            
  | 19:04 | <mutante> | rebooting hydrogen | [production] | 
            
  | 19:04 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:306255|Fix UI l10n for Help page link on commons.wikimedia.org (T143564)]] (duration: 00m 47s) | [production] | 
            
  | 18:56 | <bblack@palladium> | conftool action : set/pooled=no; selector: dc=eqiad,cluster=dns,name=hydrogen.wikimedia.org | [production] | 
            
  | 18:26 | <thcipriani@tin> | Synchronized php-1.28.0-wmf.15/resources/src/mediawiki.widgets/mw.widgets.CategoryCapsuleItemWidget.js: SWAT: [[gerrit:306234|Debug logging for "queue[title] undefined" (T139130)]] (duration: 00m 50s) | [production] | 
            
  | 18:24 | <thcipriani@tin> | Synchronized php-1.28.0-wmf.16/resources/src/mediawiki.widgets/mw.widgets.CategoryCapsuleItemWidget.js: SWAT: [[gerrit:306235|Debug logging for "queue[title] undefined" (T139130)]] (duration: 00m 50s) | [production] | 
            
  | 18:22 | <mutante> | all mw appservers are installing all the font packages now, not just imagescalers. this should fix some issues with EasyTimeline on zh projects and more | [production] | 
            
  | 18:13 | <kaldari> | mwscript maintenance/updateCollation.php --wiki=svwiki --force | [production] | 
            
  | 18:12 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:306216|Switching Swedish Wikipedia to uca-sv-u-kn collation (T142113)]] (duration: 00m 58s) | [production] | 
            
  | 17:35 | <mobrovac> | changeprop deploying 519ad9d | [production] | 
            
  | 16:27 | <jynus> | rebooting es2004 for hardware maintenance T143220 | [production] | 
            
  | 14:25 | <moritzm> | installing libgd security updates | [production] | 
            
  | 14:13 | <moritzm> | rebooting conf1002 for kernel update | [production] | 
            
  | 14:10 | <gehel> | cleanup list of banned node from elasticsearch eqiad cluster | [production] | 
            
  | 14:05 | <hashar@tin> | Finished scap: testwiki to php-1.28.0-wmf.16 T141551 (duration: 28m 45s) | [production] | 
            
  | 13:48 | <moritzm> | rebooting conf1001 for kernel update | [production] | 
            
  | 13:37 | <Dereckson> | Run namespaceDupes maintenance script on azwiki and azwiktionary (T143580) | [production] | 
            
  | 13:37 | <godog> | remove some old compilations from puppet compiler filling up compiler02.eqiad.wmflabs disks T143671 | [production] | 
            
  | 13:36 | <hashar@tin> | Started scap: testwiki to php-1.28.0-wmf.16 T141551 | [production] | 
            
  | 13:34 | <hashar> | Restarting Jenkins | [production] | 
            
  | 13:28 | <hashar> | European SWAT deploy is complete | [production] |