| 2017-02-02
      
      ยง | 
    
  | 18:03 | <bsitzmann@tin> | Started deploy [mobileapps/deploy@09101f7]: try previous deploy again (at least on canary) | [production] | 
            
  | 18:01 | <bearND> | ^ reverted previous deploy due to incorrect links in the news endpoint | [production] | 
            
  | 18:00 | <bsitzmann@tin> | Finished deploy [mobileapps/deploy@09101f7]: (no justification provided) (duration: 01m 56s) | [production] | 
            
  | 17:58 | <bsitzmann@tin> | Started deploy [mobileapps/deploy@09101f7]: (no justification provided) | [production] | 
            
  | 17:56 | <jynus> | upgrade & restart of db2059 T111654 | [production] | 
            
  | 17:43 | <jynus> | upgrade & restart of db2052 T111654 | [production] | 
            
  | 17:32 | <mobrovac> | restbase deploy end of 634faea2 | [production] | 
            
  | 17:08 | <mobrovac> | restbase deploying 634faea2 | [production] | 
            
  | 16:56 | <reedy@tin> | Synchronized php-1.29.0-wmf.10/extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php: Fix inclusion path (duration: 00m 41s) | [production] | 
            
  | 16:24 | <elukey> | reboot mc2019->mc2025 to see if they come up cleanly (currently codfw replicas of eqiad redis shards) | [production] | 
            
  | 16:13 | <elukey> | rebooting mc202[6789] (not serving any traffic) to see if they come up cleanly | [production] | 
            
  | 16:00 | <elukey> | rebooting mc203[01234] (not serving any traffic) to see if they come up cleanly | [production] | 
            
  | 15:43 | <moritzm> | upgrading firejail on remaining app servers | [production] | 
            
  | 15:35 | <moritzm> | upgrading firejail on wtp1001 | [production] | 
            
  | 15:15 | <moritzm> | upgrading firejail on eqiad imagescalers | [production] | 
            
  | 15:11 | <elukey> | rebooting mc203[56] (not taking any traffic) to test if they come up cleanly | [production] | 
            
  | 15:01 | <elukey> | Replace Redis/Memcached shards mc200[4567] with mc202[2345] | [production] | 
            
  | 14:51 | <godog> | manually fail sdc on graphite1001 - T157022 | [production] | 
            
  | 14:23 | <addshore> | EU SWAT really finished | [production] | 
            
  | 14:23 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:334997|Create Wikiprojekti namespace on Finnish Wikipedia]] T156621 (duration: 00m 41s) | [production] | 
            
  | 14:14 | <addshore> | EU SWAT finished | [production] | 
            
  | 14:14 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:335481|Add Wikinews languages (en, pt, ca, fr, de, it) as import sources on eswikinews]] T156737 (duration: 00m 40s) | [production] | 
            
  | 14:10 | <addshore@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:324489|Enable ElectronPdfService extension on dewiki]] T150942 (duration: 00m 41s) | [production] | 
            
  | 12:53 | <gehel> | starting reimage to jessie of elasticsearch relforge - T151326 | [production] | 
            
  | 11:59 | <joal@tin> | Finished deploy [analytics/refinery@bc4b4ed]: (no justification provided) (duration: 01m 14s) | [production] | 
            
  | 11:58 | <joal@tin> | Started deploy [analytics/refinery@bc4b4ed]: (no justification provided) | [production] | 
            
  | 11:40 | <elukey> | Swap mc2002 with mc2020, mc2003 with mc2021 (Redis codfw replicas) - T155755 | [production] | 
            
  | 11:35 | <joal@tin> | Finished deploy [analytics/refinery@bc4b4ed]: (no justification provided) (duration: 03m 03s) | [production] | 
            
  | 11:32 | <joal@tin> | Started deploy [analytics/refinery@bc4b4ed]: (no justification provided) | [production] | 
            
  | 10:53 | <elukey> | Swap mc2001 with mc2019 (Redis codfw replicas) - T155755 | [production] | 
            
  | 10:34 | <moritzm> | restarted hadoop-mapreduce-historyserver on analytics1001 | [production] | 
            
  | 10:23 | <moritzm> | rolling restart of nginx tls terminators running on mw* application servers in eqiad to pick up openssl 1.1 update | [production] | 
            
  | 09:54 | <moritzm> | rolling restart of logstash cluster to pick up openjdk/NSS security updates | [production] | 
            
  | 09:19 | <jynus> | deploying schema change to page_assessments_projects on enwikivoyage T156305 | [production] | 
            
  | 09:18 | <moritzm> | uograding remaining canary servers to new HHVM packages | [production] | 
            
  | 09:17 | <marostegui> | Remove dbstore1001:/srv/tmp/db1063.tar.gz after it has been transferred to db1095:/srv/tmp/db1063.tar.gz to get more disk space | [production] | 
            
  | 08:57 | <jynus> | deploying schema change to page_assessments_projects on enwiki T156305 | [production] | 
            
  | 08:42 | <filippo@tin> | Finished deploy [prometheus/jmx_exporter@23a8f0b]: jmx_exporter deploy (duration: 00m 04s) | [production] | 
            
  | 08:42 | <filippo@tin> | Started deploy [prometheus/jmx_exporter@23a8f0b]: jmx_exporter deploy | [production] | 
            
  | 08:30 | <moritzm> | installing ntfs-3g security update on labnodepool (other servers had it deinstalled) | [production] | 
            
  | 08:26 | <filippo@tin> | Finished deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided) (duration: 00m 07s) | [production] | 
            
  | 08:26 | <filippo@tin> | Started deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided) | [production] | 
            
  | 08:24 | <marostegui> | Transfer /srv/tmp/db1063.tar.gz from dbstore1001 to db1095:/srv/tmp to gain disk space | [production] | 
            
  | 08:24 | <marostegui> | Remove /srv/tmp/db1067.tar.gz from dbstore1001 to gain disk space | [production] | 
            
  | 08:23 | <filippo@tin> | Finished deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided) (duration: 00m 12s) | [production] | 
            
  | 08:23 | <filippo@tin> | Started deploy [prometheus/jmx_exporter@23a8f0b]: (no justification provided) | [production] | 
            
  | 08:10 | <legoktm@tin> | Synchronized php-1.29.0-wmf.10/includes/specials/pagers/ActiveUsersPager.php: Make last remaining user_groups queries honor $wgDisableUserGroupExpiry https://gerrit.wikimedia.org/r/#/c/335587/ (T156995) (duration: 00m 51s) | [production] | 
            
  | 08:08 | <legoktm@tin> | Synchronized php-1.29.0-wmf.10/includes/api/ApiQueryAllUsers.php: Make last remaining user_groups queries honor $wgDisableUserGroupExpiry https://gerrit.wikimedia.org/r/#/c/335587/ (T156995) (duration: 00m 58s) | [production] | 
            
  | 07:41 | <marostegui> | Restart MySQL on db2012 to tune some innodb_ft flags - T156905 | [production] | 
            
  | 03:24 | <eileen1> | renable all other jenkins jobs - only some dedupe & one-off jobs disabled | [production] |