| 2016-09-22
      
      ยง | 
    
  | 21:02 | <mutante> | imported nodepool_0.1.1-wmf5_amd64 into jessie-wikimedia (T145142) | [production] | 
            
  | 20:52 | <urandom> | T133395: RESTBase Staging: starting dumps (3, eqiad) | [production] | 
            
  | 20:47 | <urandom> | T133395: RESTBase Staging: altering table to set TWCS on wikipedia parsoid.html table | [production] | 
            
  | 20:20 | <urandom> | T133395: RESTBase Staging: Restarting Cassandra to pick up TWCS jar in classpath | [production] | 
            
  | 20:08 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.20 | [production] | 
            
  | 20:00 | <thcipriani> | rolling out wmf.20 to all wikis | [production] | 
            
  | 19:23 | <SMalyshev> | Deploying new version of WDQS GUI | [production] | 
            
  | 19:09 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.20 | [production] | 
            
  | 19:01 | <thcipriani> | wmf.20 to group1 will watch until 20 UTC and move forward to all wikis | [production] | 
            
  | 18:56 | <thcipriani@tin> | Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: SWAT: [[gerrit:312292|Update extensions/CentralNotice submodule (T144952)]] (duration: 00m 50s) | [production] | 
            
  | 18:50 | <yuvipanda> | enabling puppet on labcontrol1001, run on labtestcontrol2001 seems ok | [production] | 
            
  | 18:47 | <thcipriani@tin> | Synchronized php-1.28.0-wmf.20/extensions/CentralNotice: SWAT: [[gerrit:312293|Update extensions/CentralNotice submodule (T144952)]] (duration: 00m 52s) | [production] | 
            
  | 18:46 | <yuvipanda> | disable puppet on labcontrol1001 for https://gerrit.wikimedia.org/r/#/c/312301/ | [production] | 
            
  | 18:38 | <aaron@tin> | Synchronized php-1.28.0-wmf.20/includes/libs/rdbms/database/Database.php: 844cfd568a7c7953faa6ac69acebff1cee943b7f & 014a420b4525798b1202cc488b337acdaf09c49a (duration: 00m 49s) | [production] | 
            
  | 18:31 | <thcipriani@tin> | Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler/MwEmbedModules: SWAT: [[gerrit:312264|Update ogv.js to 1.2.0 (T145983)]] (duration: 00m 48s) | [production] | 
            
  | 18:28 | <thcipriani@tin> | Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler/MwEmbedModules: SWAT: [[gerrit:312263|Update ogv.js to 1.2.0 (T145983)]] (duration: 00m 51s) | [production] | 
            
  | 17:24 | <moritzm> | rebooting ms-be1016, high load caused by XFS bug | [production] | 
            
  | 17:09 | <moritzm> | rolling reboot of trusty swift backend servers in eqiad completed | [production] | 
            
  | 16:40 | <elukey> | forced logrotation for /etc/logrotate.d/upstart on labvirt1014 to investigate cronspam | [production] | 
            
  | 16:17 | <godog> | offline sdd on ms-be1004 via megacli T144499 | [production] | 
            
  | 15:29 | <mobrovac> | restbase deploy end of d96fbc1 | [production] | 
            
  | 15:10 | <mobrovac> | restbase deploy start of d96fbc1 | [production] | 
            
  | 15:02 | <bblack> | upgrading openssl on cp* | [production] | 
            
  | 13:22 | <moritzm> | resume rolling reboot of trusty swift backend servers in eqiad for kernel security update | [production] | 
            
  | 13:02 | <moritzm> | uploaded openssl 1.0.2i for jessie-wikimedia to carbon | [production] | 
            
  | 12:31 | <hashar@tin> | Synchronized php-1.28.0-wmf.20/extensions/Popups: Merge mw.popups.experiment into mw.popups.core T146035 (duration: 00m 49s) | [production] | 
            
  | 12:27 | <mobrovac> | restbase deploy end of d5538ad | [production] | 
            
  | 12:25 | <elukey> | installing varnishkafka 1.0.12 on cache:upload ulsfo and eqiad | [production] | 
            
  | 12:24 | <akosiaris> | uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ro_0.7.3~r57551-2+wmf1 | [production] | 
            
  | 12:13 | <hashar> | Early SWAT for mobile team ( https://gerrit.wikimedia.org/r/#/c/311977/ ) | [production] | 
            
  | 12:11 | <mobrovac> | restbase deploy start of d5538ad | [production] | 
            
  | 11:34 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1082 with some light weight (duration: 00m 52s) | [production] | 
            
  | 10:24 | <moritzm> | rolling reboot of trusty swift backend servers in eqiad for kernel security update | [production] | 
            
  | 09:59 | <moritzm> | rebooting subra/suhail for kernel security update | [production] | 
            
  | 09:50 | <hashar> | updated jobrunner code to a0e82166 (tweak errors reporting in logs)  | Does not include 51014242 "Batch stats to statsd"  (poke addshore ) | [production] | 
            
  | 09:19 | <gehel> | upgrade / restart of elasticsearch eqiad cluster done T145404 / T146123 | [production] | 
            
  | 09:02 | <elukey> | installing varnishkafka 1.0.12 on cache:upload codfw | [production] | 
            
  | 08:49 | <marostegui> | Deploying schema change on S7 master - T141951 | [production] | 
            
  | 08:43 | <elukey> | installing varnishkafka 1.0.12 on cache:upload esams | [production] | 
            
  | 08:40 | <elukey> | installed varnishkafka 1.0.12 on cp1099 | [production] | 
            
  | 08:35 | <elukey> | restarted varnishkafka on cp1099 (log abandoned ) | [production] | 
            
  | 08:19 | <hashar> | Cleanup jobrunner list of minions in redis ( "deploy:jobrunner/jobrunner:minions" ) | [production] | 
            
  | 08:09 | <hashar> | Resyncing all jobrunner deployment installations since only 41/68 minions have completed fetch/checkout | [production] | 
            
  | 08:01 | <elukey> | rolling restart of the whole Analytics Hadoop cluster for kernel upgrades (analytics* hosts) | [production] | 
            
  | 07:58 | <elukey> | uploaded varnishkafka 1.0.12-1 to reprepro | [production] | 
            
  | 07:52 | <elukey> | rebooted stat100[23] for kernel upgrades | [production] | 
            
  | 07:40 | <moritzm> | rolling restart of trusty swift frontend servers in codfw for kernel security update | [production] | 
            
  | 07:33 | <elukey> | rebooting stat1004 for kernel upgrades | [production] | 
            
  | 06:45 | <elukey> | Puppet disabled on analytics1027 to stop periodic Java daemons (prep step for Hadoop cluster reboots) | [production] | 
            
  | 03:18 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Thu Sep 22 03:18:48 UTC 2016 (duration 6m 48s) | [production] |