| 2016-05-03
      
      § | 
    
  | 12:07 | <hashar> | Cutting 1.27.0-wmf.23 branches | [production] | 
            
  | 11:35 | <bblack> | cp1068: upgraded varnish to 3.0.6plus-wm9 | [production] | 
            
  | 10:42 | <moritzm> | installing poppler security updates on ocg (and other trusty hosts) | [production] | 
            
  | 10:24 | <jynus> | applying schema change on eqiad-s3 db servers T130692 | [production] | 
            
  | 10:03 | <jynus> | applying schema change on codfw-s3 db servers T130692 | [production] | 
            
  | 09:55 | <jynus> | testing online index creation on db2018 | [production] | 
            
  | 09:11 | <gehel> | restarting elasticsearch server elastic2003.codfw.wmnet (T110236) | [production] | 
            
  | 09:10 | <moritzm> | powercycled alsafi (stuck in KVM) | [production] | 
            
  | 09:08 | <kart_> | Update cxserver to 8a4254e | [production] | 
            
  | 08:18 | <gehel> | restarting elasticsearch server elastic2002.codfw.wmnet (T110236) | [production] | 
            
  | 08:05 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1040 after maintenance (duration: 00m 25s) | [production] | 
            
  | 07:06 | <jynus> | recovering torrus database @ netmon following T87815 | [production] | 
            
  | 02:34 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Tue May  3 02:34:45 UTC 2016 (duration 8m 36s) | [production] | 
            
  | 02:26 | <awight> | CentralNotice fundraising campaigns reenabled after CiviCRM maintenance | [production] | 
            
  | 02:26 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.22) (duration: 08m 40s) | [production] | 
            
  | 02:21 | <mutante> | ssh alsafi | [production] | 
            
  | 02:13 | <eileen> | Updating civicrm from f5e8f98d07a2280118b7153bc342bf52ee67edd5 to d32032965f2a45d24f1eaeb352eb6bfd6641753c | [production] | 
            
  | 02:05 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Remove obsolete https->http rewrite for IRC notifications (duration: 00m 28s) | [production] | 
            
  | 02:04 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Remove obsolete ocwiki hack (duration: 00m 37s) | [production] | 
            
  | 01:54 | <RoanKattouw> | Killed ssh processes on tin that had been hanging for days | [production] | 
            
  | 01:53 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.21) (duration: 23019m 56s) | [production] | 
            
  | 01:53 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.21) (duration: 12941m 41s) | [production] | 
            
  | 01:53 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.22) (duration: 4303m 19s) | [production] | 
            
  | 01:40 | <mutante> | irc.wm.org - see T123729 if any questions | [production] | 
            
  | 01:39 | <mutante> | switching irc.wikimedia.org from old server argon to new server kraz. old server still running untouched as argon.wikimedia.org. no clients are kicked. appservers are sending RC to both. | [production] | 
            
  | 00:50 | <krenair@tin> | Synchronized wmf-config/interwiki.php: https://gerrit.wikimedia.org/r/286552 - interwiki cache update including jamwiki, horizon, and other things (duration: 00m 26s) | [production] | 
            
  | 00:41 | <krenair@tin> | Synchronized langlist: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s) | [production] | 
            
  | 00:40 | <krenair@tin> | Synchronized static/images/project-logos: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s) | [production] | 
            
  | 00:39 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 24s) | [production] | 
            
  | 00:38 | <krenair@tin> | rebuilt wikiversions.php and synchronized wikiversions files: create jamwiki - https://gerrit.wikimedia.org/r/286258 | [production] | 
            
  | 00:36 | <krenair@tin> | Synchronized dblists: create jamwiki - https://gerrit.wikimedia.org/r/286258 (duration: 00m 27s) | [production] | 
            
  | 00:12 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/286534/ (duration: 00m 30s) | [production] | 
            
  
    | 2016-05-02
      
      § | 
    
  | 23:58 | <krenair@tin> | Finished scap: for https://gerrit.wikimedia.org/r/#/c/286535/ i18n changes (duration: 25m 57s) | [production] | 
            
  | 23:37 | <ebernhardson> | restart elastic2007, codfw cluster master, to resolve lingering issues after resolving frozen index race condition | [production] | 
            
  | 23:32 | <krenair@tin> | Started scap: for https://gerrit.wikimedia.org/r/#/c/286535/ i18n changes | [production] | 
            
  | 23:30 | <krenair@tin> | Synchronized php-1.27.0-wmf.22/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/286535/ and https://gerrit.wikimedia.org/r/#/c/286532/ (duration: 00m 30s) | [production] | 
            
  | 23:10 | <krenair@tin> | Synchronized php-1.27.0-wmf.22/extensions/Flow/modules/flow/ui/widgets/editor/mw.flow.ui.EditorSwitcherWidget.js: https://gerrit.wikimedia.org/r/#/c/286527/ (duration: 00m 26s) | [production] | 
            
  | 22:56 | <ebernhardson@tin> | Synchronized php-1.27.0-wmf.22/extensions/CirrusSearch/: Cirrus: Stop auto-creating frozen index (duration: 00m 31s) | [production] | 
            
  | 22:56 | <awight> | CentralNotice fundraising campaigns disabled for CiviCRM outage | [production] | 
            
  | 22:52 | <ebernhardson@tin> | Synchronized wmf-config/CirrusSearch-production.php: Config setting to stop auto-creating frozen index in cirrus (duration: 00m 33s) | [production] | 
            
  | 22:18 | <krenair@tin> | Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/286537/ (duration: 00m 42s) | [production] | 
            
  | 22:13 | <cwd> | updated crm from b386a6821c71310950ccdcdcf2616add727e1af4 to f5e8f98d07a2280118b7153bc342bf52ee67edd5 | [production] | 
            
  | 22:10 | <papaul> | restbase200[7-9]- signing puppet certs, salt-key, initial run | [production] | 
            
  | 21:26 | <cscott> | updated OCG to version b775e612520f9cd4acaea42226bcf34df07439f7 | [production] | 
            
  | 21:21 | <cscott> | starting OCG deploy (a little late) | [production] | 
            
  | 20:23 | <gehel> | restarting elasticsearch server elastic2001.codfw.wmnet (T110236) | [production] | 
            
  | 20:21 | <gehel> | starting rolling restart of elasticsearch codfw cluster to disable multicast (T110236) | [production] | 
            
  | 20:15 | <subbu> | finished deploying parsoid version 0a26f3a4 | [production] | 
            
  | 20:09 | <subbu> | synced code + restarted parsoid on wtp1001 as canary | [production] | 
            
  | 20:05 | <subbu> | starting deploy of parsoid version 0a26f3a4 | [production] |