| 2016-01-28
      
      ยง | 
    
  | 23:51 | <mutante> | caesium - stop puppet, shutdown server, remove from icinga, clean puppet cert ... | [production] | 
            
  | 23:46 | <TimStarling> | on ruthenium installing build dependencies and compiling uprightdiff for test | [production] | 
            
  | 23:20 | <ori@mira> | Synchronized php-1.27.0-wmf.11/includes/api/ApiStashEdit.php: Ia4196eba9: Add ParserOutputStashForEdit hook for extension cache warming (duration: 01m 10s) | [production] | 
            
  | 23:17 | <tgr@mira> | Synchronized php-1.27.0-wmf.11/includes/session/SessionManager.php: T125161 (duration: 01m 11s) | [production] | 
            
  | 22:58 | <ottomata> | restoring MobileWebSectionUsage_14321266 from db1047 to dbstore1002 using mysqlimport | [production] | 
            
  | 22:23 | <bblack> | starting cache_mobile->cache_text conversion in eqiad - https://phabricator.wikimedia.org/T109286 | [production] | 
            
  | 22:09 | <bblack> | eqiad pybal->etcd conversion done | [production] | 
            
  | 22:01 | <dduvall@mira> | Synchronized php-1.27.0-wmf.11/extensions/WikimediaEvents/WikimediaEventsHooks.php: deploying fix for T125151 (duration: 01m 15s) | [production] | 
            
  | 21:59 | <mutante> | releases.wm.org - switched backend to bromine | [production] | 
            
  | 21:58 | <bblack> | converting active eqiad LVS/pybal to etcd | [production] | 
            
  | 21:55 | <mutante> | caesium - stopped apache | [production] | 
            
  | 21:31 | <ori@mira> | Synchronized php-1.27.0-wmf.11/extensions/AbuseFilter: I13fcc3ce4: Updated mediawiki/core Project: mediawiki/extensions/AbuseFilter  19baa3b6e51b8fe6baf6e3ce7e590060e8e6eec9 (duration: 01m 11s) | [production] | 
            
  | 21:27 | <bblack> | converting backup/inactive eqiad LVS/pybal to etcd | [production] | 
            
  | 21:16 | <dduvall@mira> | rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.11 | [production] | 
            
  | 20:54 | <mutante> | sca1001 - stop mathoid,graphoid,citoid | [production] | 
            
  | 20:52 | <mutante> | sca1002 - stop mathoid,graphoid,citoid | [production] | 
            
  | 20:50 | <dduvall@mira> | Synchronized php-1.27.0-wmf.11: syncing 1.27.0-wmf.11 for T125114 and https://gerrit.wikimedia.org/r/#/c/267128/ (duration: 03m 30s) | [production] | 
            
  | 20:25 | <bblack> | depool -> reboot cp4008 (ulsfo text, trying new kernel with live traffic) | [production] | 
            
  | 20:00 | <bblack> | depool -> reboot cp4011 (ulsfo mobile, currently unused for traffic - testing local conftool-scripts depool + new kernel) | [production] | 
            
  | 19:55 | <ori@mira> | Synchronized wmf-config: Iea2573ccfbe: Revert "Autopromotion: remove deprecated onView event, fix INGROUPS" (duration: 02m 13s) | [production] | 
            
  | 19:43 | <ori> | added tgr and marxarelli to security group on phab | [production] | 
            
  | 19:26 | <ottomata> | kafka preferred-replica-election to rebalanace analytics-eqiad brokers | [production] | 
            
  | 18:22 | <elukey> | rebooting analytics1001 for new kernel upgrade | [production] | 
            
  | 18:21 | <yurik> | deployed graphoid | [production] | 
            
  | 17:43 | <elukey> | rebooting analytics1002.eqiad.wmnet (Hadoop master's slave) for kernel upgrade | [production] | 
            
  | 17:39 | <urandom> | finished deploying configuration change (https://gerrit.wikimedia.org/r/266299) to restbase staging | [production] | 
            
  | 17:38 | <robh> | neglected to log i ifinished icinga/neon updates and its back to normal service (never interrrupted) | [production] | 
            
  | 17:38 | <urandom> | restarting restbase on restbase200[1-3].codfw.wmnet (restbase staging) | [production] | 
            
  | 17:34 | <urandom> | forcing puppet run on restbase200[1-3].codfw.wmnet (restbase staging) | [production] | 
            
  | 17:30 | <urandom> | forcing puppet run on praseodymium.eqiad.wmnet, and restarting restbase (staging env) | [production] | 
            
  | 17:27 | <urandom> | restarting restbase on xenon.eqiad.wmnet (restbase staging) | [production] | 
            
  | 17:25 | <urandom> | forcing puppet run on xenon.eqiad.wmnet (restbase staging) | [production] | 
            
  | 17:21 | <urandom> | restarting restbase on cerium.eqiad.wmnet | [production] | 
            
  | 17:18 | <urandom> | forcing puppet run on cerium.eqiad.wmnet (restbase staging) | [production] | 
            
  | 17:18 | <robh> | pushing icinga updates (shouldnt affect service but others shouldnt also try to update neon right now) | [production] | 
            
  | 17:17 | <krenair@mira> | Synchronized README: testing (duration: 02m 08s) | [production] | 
            
  | 17:15 | <urandom> | disabling pupplet on restbase staging hosts | [production] | 
            
  | 17:01 | <krenair@mira> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266957/ (duration: 02m 15s) | [production] | 
            
  | 16:51 | <krenair@mira> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/267040/ (duration: 02m 13s) | [production] | 
            
  | 16:48 | <cmjohnson1> | mw1172, mw1178,mw1217, mw1257 powering off task# T124642 | [production] | 
            
  | 16:45 | <krenair@mira> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264219/ (duration: 02m 12s) | [production] | 
            
  | 16:42 | <krenair@mira> | Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/264219/ (duration: 02m 12s) | [production] | 
            
  | 16:37 | <Krenair> | Downloaded and `chmod +x`'d mira:/srv/mediawiki-staging/.git/hooks/commit-msg | [production] | 
            
  | 16:29 | <mdholloway> | mobileapps deployed 7583148, reverting in part 869ec35 | [production] | 
            
  | 16:25 | <krenair@mira> | Synchronized wmf-config/InitialiseSettings.php: rv (duration: 02m 10s) | [production] | 
            
  | 16:25 | <bblack> | upgrading packages (incl kernel) on all codfw caches | [production] | 
            
  | 16:19 | <krenair@mira> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266955/ (duration: 02m 14s) | [production] | 
            
  | 16:13 | <krenair@mira> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/266564/ (duration: 02m 12s) | [production] | 
            
  | 16:04 | <krenair@mira> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264733/ (duration: 02m 11s) | [production] | 
            
  | 15:39 | <bblack> | kafka1012 booted up normally | [production] |