| 2018-10-04
      
      § | 
    
  | 12:01 | <moritzm> | rolling reboot of ms-fe hosts in codfw for kernel security update | [production] | 
            
  | 12:00 | <zeljkof> | one more patch for EU SWAT | [production] | 
            
  | 11:57 | <zeljkof> | EU SWAT finished | [production] | 
            
  | 11:57 | <zfilipin@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:460700|Add *.nasimonline.ir to wgCopyUploadsDomains whitelist for Commons (T203371)]] (duration: 00m 56s) | [production] | 
            
  | 11:52 | <zfilipin@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:457474|add Radlines.org to $wgCopyUploadsDomains (T203219)]] (duration: 00m 57s) | [production] | 
            
  | 11:42 | <zfilipin@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:457469|Add .bollywoodhungama.in to wgCopyUploadsDomains (T203363)]] (duration: 00m 57s) | [production] | 
            
  | 11:35 | <zfilipin@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:463227|Add some namespaces aliases for zhwikiversity (T201675)]] (duration: 00m 57s) | [production] | 
            
  | 11:27 | <zfilipin@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:463584|Change acewiki default time zone to Asia/Jakarta (T205693)]] (duration: 00m 56s) | [production] | 
            
  | 11:17 | <zfilipin@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:463582|Create Photowalk and Photowalk Talk namespaces for bd.wikimedia.org (T205747)]] (duration: 00m 57s) | [production] | 
            
  | 10:44 | <twentyafterfour@deploy1001> | Synchronized php-1.32.0-wmf.23/README: noop sync to verify that scap 3.8.7-1 works (at least on a basic level) (duration: 00m 59s) | [production] | 
            
  | 10:38 | <godog> | upload scap 3.8.7-1 - T204383 | [production] | 
            
  | 10:36 | <_joe_> | uploading etcd-mirror to stretch-wikimedia T205814 | [production] | 
            
  | 10:08 | <moritzm> | rolling reboot of ms-fe hosts in eqiad for kernel security update | [production] | 
            
  | 09:13 | <arturo> | T203177 schedule 8h icinga downtime for cloudcontrol1003,1004 and labmon1001 | [production] | 
            
  | 08:52 | <moritzm> | installing python2.7/python3.4/python3.5 security updates on jessie/stretch | [production] | 
            
  | 08:34 | <moritzm> | installing ca-certificates updates for jessie/stretch | [production] | 
            
  | 08:09 | <marostegui> | Restart icinga T196336 | [production] | 
            
  | 08:00 | <gehel> | re-enabling puppet on maps1004 | [production] | 
            
  | 07:31 | <elukey> | move Piwik/Matomo from bohrium to matomo1001 - T202962 | [production] | 
            
  | 07:25 | <godog> | reformat ms-be1041 with crc=1 finobt=0 - T199198 | [production] | 
            
  | 06:57 | <jynus> | starting multisource replication of s3 from s5 at eqiad master | [production] | 
            
  | 06:51 | <jynus> | reenabling consistency configuration on s5 replica databases | [production] | 
            
  | 06:24 | <jynus> | create manual backup of databases on eqiad s6, s7, s8, x1 | [production] | 
            
  | 05:36 | <marostegui> | Deploy schema change on db2048 (s1 master) - T205913 | [production] | 
            
  | 05:35 | <marostegui@deploy1001> | Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 56s) | [production] | 
            
  | 05:30 | <marostegui> | Deploy schema change on db2062 - T205913 | [production] | 
            
  | 05:30 | <marostegui@deploy1001> | Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 57s) | [production] | 
            
  | 04:04 | <SMalyshev> | repooled wdqs2003 | [production] | 
            
  | 03:22 | <SMalyshev> | depool wdqs2003 to let it catch up | [production] | 
            
  | 03:21 | <SMalyshev> | repooled wdqs2001 | [production] | 
            
  | 03:16 | <ejegg> | re-enabled PayPal EC orphan rectifier | [production] | 
            
  | 03:06 | <ejegg> | updated CiviCRM from 80cb98e33e to ddf4865650 | [production] | 
            
  | 02:43 | <SMalyshev> | depooled wdqs2001 to see if it catches up faster | [production] | 
            
  | 01:54 | <ejegg> | updated payments-wiki from 8b673cfb4f to d623de9494 | [production] | 
            
  
    | 2018-10-03
      
      § | 
    
  | 23:54 | <mutante> | scheduled downtime for wdqs as it's flapping and already known | [production] | 
            
  | 23:44 | <catrope@deploy1001> | Synchronized php-1.32.0-wmf.23/extensions/VisualEditor/: Require Parsoid HTML 2.0.0, and handle its <audio> tags (T201081); ext.visualEditor.mwlanguage: Actually load all of the code (T205834) (duration: 00m 57s) | [production] | 
            
  | 23:41 | <catrope@deploy1001> | Synchronized php-1.32.0-wmf.24/extensions/VisualEditor/: Require Parsoid HTML 2.0.0, and handle its <audio> tags (T201081) (duration: 00m 59s) | [production] | 
            
  | 23:29 | <catrope@deploy1001> | Synchronized php-1.32.0-wmf.23/extensions/PageTriage/: Hide copyvio AFC filter option behind flag (T205918) (duration: 00m 57s) | [production] | 
            
  | 23:23 | <catrope@deploy1001> | Synchronized php-1.32.0-wmf.24/includes/utils/UIDGenerator.php: Make UID clock drift error have more details (T94522) (duration: 00m 58s) | [production] | 
            
  | 23:20 | <XenoRyet> | shut off Paypal orphan rectifier | [production] | 
            
  | 23:12 | <catrope@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Bump Minerva A/B test rates to 100% on jawiki, ruwiki, fawiki (T200792) (duration: 00m 56s) | [production] | 
            
  | 22:49 | <shdubsh> | re-enable puppet on einsteinium | [production] | 
            
  | 22:45 | <shdubsh> | einsteinium: setting enable_notifications=1 and reloading icinga | [production] | 
            
  | 22:36 | <herron> | herron@neodymium:~$ sudo cumin -b 15 -p 95 '*' 'run-puppet-agent -q --failed-only' | [production] | 
            
  | 22:20 | <shdubsh> | einsteinium: setting enable_notifications=0 and starting icinga | [production] | 
            
  | 22:06 | <herron> | herron@neodymium:~$ sudo cumin -b 40 -p 95 'R:file = /etc/nagios/nrpe_local.cfg' run-puppet-agent | [production] | 
            
  | 22:02 | <mutante> | mw2242 - started nagios-nrpe-server | [production] | 
            
  | 22:01 | <shdubsh> | icinga stopped manually | [production] | 
            
  | 21:57 | <mutante> | einstienium - disabling puppet | [production] | 
            
  | 21:25 | <bblack> | upgraded gdnsd to 2.99.9161 on authdns1001 | [production] |