| 
      
        2016-09-22
      
      §
     | 
  
    
  | 09:59 | 
  <moritzm> | 
  rebooting subra/suhail for kernel security update | 
  [production] | 
            
  | 09:50 | 
  <hashar> | 
  updated jobrunner code to a0e82166 (tweak errors reporting in logs)  | Does not include 51014242 "Batch stats to statsd"  (poke addshore ) | 
  [production] | 
            
  | 09:19 | 
  <gehel> | 
  upgrade / restart of elasticsearch eqiad cluster done T145404 / T146123 | 
  [production] | 
            
  | 09:02 | 
  <elukey> | 
  installing varnishkafka 1.0.12 on cache:upload codfw | 
  [production] | 
            
  | 08:49 | 
  <marostegui> | 
  Deploying schema change on S7 master - T141951 | 
  [production] | 
            
  | 08:43 | 
  <elukey> | 
  installing varnishkafka 1.0.12 on cache:upload esams | 
  [production] | 
            
  | 08:40 | 
  <elukey> | 
  installed varnishkafka 1.0.12 on cp1099 | 
  [production] | 
            
  | 08:35 | 
  <elukey> | 
  restarted varnishkafka on cp1099 (log abandoned ) | 
  [production] | 
            
  | 08:19 | 
  <hashar> | 
  Cleanup jobrunner list of minions in redis ( "deploy:jobrunner/jobrunner:minions" ) | 
  [production] | 
            
  | 08:09 | 
  <hashar> | 
  Resyncing all jobrunner deployment installations since only 41/68 minions have completed fetch/checkout | 
  [production] | 
            
  | 08:01 | 
  <elukey> | 
  rolling restart of the whole Analytics Hadoop cluster for kernel upgrades (analytics* hosts) | 
  [production] | 
            
  | 07:58 | 
  <elukey> | 
  uploaded varnishkafka 1.0.12-1 to reprepro | 
  [production] | 
            
  | 07:52 | 
  <elukey> | 
  rebooted stat100[23] for kernel upgrades | 
  [production] | 
            
  | 07:40 | 
  <moritzm> | 
  rolling restart of trusty swift frontend servers in codfw for kernel security update | 
  [production] | 
            
  | 07:33 | 
  <elukey> | 
  rebooting stat1004 for kernel upgrades | 
  [production] | 
            
  | 06:45 | 
  <elukey> | 
  Puppet disabled on analytics1027 to stop periodic Java daemons (prep step for Hadoop cluster reboots) | 
  [production] | 
            
  | 03:18 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Thu Sep 22 03:18:48 UTC 2016 (duration 6m 48s) | 
  [production] | 
            
  | 03:12 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.20) (duration: 17m 08s) | 
  [production] | 
            
  | 02:38 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 39s) | 
  [production] | 
            
  | 02:03 | 
  <eileen> | 
  all back on | 
  [production] | 
            
  | 01:50 | 
  <eileen> | 
  turned off jenkins jobs to run dbupdate: dedupe(s) thank-you & donate import | 
  [production] | 
            
  | 01:40 | 
  <eileen> | 
  update civicrm from 5393a13727e8dbad05ffef9ddd44e965eb9282d1 to d30a5e454e6614214faef8354a3a23d71d573c7f | 
  [production] | 
            
  | 01:35 | 
  <twentyafterfour> | 
  reboot successful, iridium is back online | 
  [production] | 
            
  | 01:28 | 
  <twentyafterfour> | 
  Rebooting iridium to apply kernel update | 
  [production] | 
            
  | 00:39 | 
  <awight> | 
  update paymentswiki from d572ee9e3ef0a97c044f02e9866469a8f3fa5858 to cc27f83f31ecc609d4400050e73905b7364f1d42; mirror unsubscribe queue | 
  [production] | 
            
  
    | 
      
        2016-09-21
      
      §
     | 
  
    
  | 23:32 | 
  <dereckson@tin> | 
  Synchronized wmf-config/CommonSettings.php: Blacklist minerva from showing Related Articles in the footer (T144912, currently no-op) (duration: 00m 47s) | 
  [production] | 
            
  | 23:31 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Blacklist minerva from showing Related Articles in the footer (T144912, currently no-op) (duration: 00m 49s) | 
  [production] | 
            
  | 22:57 | 
  <mobrovac> | 
  change-prop deploying ea8cdf8 | 
  [production] | 
            
  | 22:35 | 
  <awight@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Add CentralNotice debug log bucket for T144952 (duration: 00m 48s) | 
  [production] | 
            
  | 22:33 | 
  <awight@tin> | 
  Synchronized php-1.28.0-wmf.20/extensions/CentralNotice: Correct CentralNotice logging for T144952 (duration: 00m 51s) | 
  [production] | 
            
  | 22:31 | 
  <awight@tin> | 
  Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: Correct CentralNotice logging for T144952 (duration: 00m 51s) | 
  [production] | 
            
  | 22:08 | 
  <cwd> | 
  updated SmashPig from f308ba490635c75b6fe735e7dc1c558250202365 to 8ac116037440746eaf64b9e99e1ee962d5d33475 | 
  [production] | 
            
  | 20:39 | 
  <bearND> | 
  deployed mobileapps bf6943b | 
  [production] | 
            
  | 20:36 | 
  <yurik> | 
  deployed kartotherian geoshape lines support - https://gerrit.wikimedia.org/r/#/c/312097/ | 
  [production] | 
            
  | 20:35 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.20 | 
  [production] | 
            
  | 20:31 | 
  <bearND> | 
  starting mobileapps deploy | 
  [production] | 
            
  | 20:24 | 
  <thcipriani@tin> | 
  Finished scap: testwiki to php-1.28.0-wmf.20 and rebuild l10n cache (duration: 52m 01s) | 
  [production] | 
            
  | 20:17 | 
  <arlolra> | 
  updated Parsoid to version a802de0 | 
  [production] | 
            
  | 20:04 | 
  <arlolra> | 
  starting Parsoid deploy | 
  [production] | 
            
  | 19:32 | 
  <thcipriani@tin> | 
  Started scap: testwiki to php-1.28.0-wmf.20 and rebuild l10n cache | 
  [production] | 
            
  | 19:24 | 
  <krinkle@tin> | 
  Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: T146099 (duration: 01m 41s) | 
  [production] | 
            
  | 17:41 | 
  <bblack> | 
  bits.wikimedia.org hostname removed from DNS (if related real complaints/problems occur, revert https://gerrit.wikimedia.org/r/305533 ) | 
  [production] | 
            
  | 17:40 | 
  <elukey> | 
  installed varnishkafka 1.0.12-1 on cp3034.esams (T138747) | 
  [production] | 
            
  | 16:46 | 
  <Krenair> | 
  running P3833 script against designate to clean up existing T120797 mess | 
  [production] | 
            
  | 16:46 | 
  <mobrovac> | 
  restbase deploy end of a75510d | 
  [production] | 
            
  | 16:30 | 
  <mobrovac> | 
  restbase deploy start of a75510d | 
  [production] | 
            
  | 15:03 | 
  <moritzm> | 
  installing wireshark security updates | 
  [production] | 
            
  | 13:58 | 
  <akosiaris> | 
  disabled puppet on neon, puppet migration in progress | 
  [production] | 
            
  | 13:48 | 
  <hashar> | 
  European SWAT completed | 
  [production] | 
            
  | 13:45 | 
  <hashar@tin> | 
  Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 46s) | 
  [production] |