| 
      
        2017-03-30
      
      §
     | 
  
    
  | 05:56 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1094 - T17441 (duration: 00m 45s) | 
  [production] | 
            
  | 05:51 | 
  <ema> | 
  upgrading twisted to 16.2.0 on lvs100[456] (eqiad secondaries) T160433 | 
  [production] | 
            
  | 03:03 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Thu Mar 30 03:03:31 UTC 2017 (duration 5m 49s) | 
  [production] | 
            
  | 02:57 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.29.0-wmf.18) (duration: 13m 43s) | 
  [production] | 
            
  | 02:22 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.29.0-wmf.17) (duration: 07m 20s) | 
  [production] | 
            
  | 01:52 | 
  <twentyafterfour> | 
  phd fixed on iridium. libphutil was out of sync with phd source | 
  [production] | 
            
  | 01:11 | 
  <twentyafterfour> | 
  running `puppet agent --test` on iridium | 
  [production] | 
            
  | 00:10 | 
  <twentyafterfour> | 
  Phabricator update completed. | 
  [production] | 
            
  | 00:10 | 
  <mutante> | 
  ruthenium low on disk space, because /srv/visualdiff/pngs (parsoid-vd-tests) is pretty large and /srv isn't a separate mount | 
  [production] | 
            
  | 00:06 | 
  <twentyafterfour> | 
  updating phabricator on iridium | 
  [production] | 
            
  | 00:04 | 
  <mutante> | 
  ruthenium - apt-get clean gets a little more disk space | 
  [production] | 
            
  
    | 
      
        2017-03-29
      
      §
     | 
  
    
  | 23:46 | 
  <reedy@tin> | 
  Synchronized php-1.29.0-wmf.18/extensions/Quiz: Fix undefined variable stateObject T161735 (duration: 00m 49s) | 
  [production] | 
            
  | 23:43 | 
  <reedy@tin> | 
  Synchronized wmf-config/CommonSettings.php: Dont use EP_NS in CommonSettings (duration: 00m 44s) | 
  [production] | 
            
  | 21:23 | 
  <krinkle@tin> | 
  Synchronized errorpages/: I15295835a1a (duration: 00m 44s) | 
  [production] | 
            
  | 20:56 | 
  <thcipriani@tin> | 
  Synchronized php-1.29.0-wmf.18/extensions/ProofreadPage/includes/page/ProofreadPagePage.php: [[gerrit:345423|Makes sure to always return a Title in ProofreadPagePage::findIndexTitle]] T161734 (duration: 00m 46s) | 
  [production] | 
            
  | 20:35 | 
  <halfak@tin> | 
  Finished deploy [ores/deploy@554ea12]: T160638 (duration: 18m 40s) | 
  [production] | 
            
  | 20:31 | 
  <arlolra> | 
  Updated Parsoid to b1b27146 (T161558, T160207, T153798) | 
  [production] | 
            
  | 20:21 | 
  <arlolra@tin> | 
  Finished deploy [parsoid/deploy@bc798dc]: Updating Parsoid to b1b27146 (duration: 07m 26s) | 
  [production] | 
            
  | 20:16 | 
  <halfak@tin> | 
  Started deploy [ores/deploy@554ea12]: T160638 | 
  [production] | 
            
  | 20:13 | 
  <arlolra@tin> | 
  Started deploy [parsoid/deploy@bc798dc]: Updating Parsoid to b1b27146 | 
  [production] | 
            
  | 20:09 | 
  <ppchelko@tin> | 
  Finished deploy [changeprop/deploy@ef62908]: Fix metrics for regex topics (duration: 00m 56s) | 
  [production] | 
            
  | 20:08 | 
  <ppchelko@tin> | 
  Started deploy [changeprop/deploy@ef62908]: Fix metrics for regex topics | 
  [production] | 
            
  | 19:46 | 
  <ppchelko@tin> | 
  Finished deploy [changeprop/deploy@1150cf5]: Config: Enabling regex-based topic subscription (duration: 01m 45s) | 
  [production] | 
            
  | 19:44 | 
  <ppchelko@tin> | 
  Started deploy [changeprop/deploy@1150cf5]: Config: Enabling regex-based topic subscription | 
  [production] | 
            
  | 19:16 | 
  <awight> | 
  re-run today's ingenico audit job | 
  [production] | 
            
  | 19:15 | 
  <awight> | 
  pick at paypal scab: re-run audit parser | 
  [production] | 
            
  | 19:10 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.29.0-wmf.17 | 
  [production] | 
            
  | 19:05 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.18 | 
  [production] | 
            
  | 18:45 | 
  <bblack> | 
  varnish active/active deploy done ( https://gerrit.wikimedia.org/r/#/c/339667/ ) - all caches running the new code, puppet re-enabled, etc. | 
  [production] | 
            
  | 18:43 | 
  <hoo> | 
  Started a Wikidata TTL dump run on snapshot1007 using Zend (due to T161695). | 
  [production] | 
            
  | 18:22 | 
  <catrope@tin> | 
  Synchronized php-1.29.0-wmf.18/includes/page/WikiPage.php: T159319 (duration: 00m 44s) | 
  [production] | 
            
  | 18:22 | 
  <catrope@tin> | 
  Synchronized php-1.29.0-wmf.18/includes/Title.php: T159319 (duration: 00m 46s) | 
  [production] | 
            
  | 18:10 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Save->Publihs on Wikipedias except dewiki and enwiki (T131132); set wgOOUIEditPage false everywhere (duration: 00m 57s) | 
  [production] | 
            
  | 17:58 | 
  <awight> | 
  disabling PayPal audit parser | 
  [production] | 
            
  | 17:56 | 
  <ppchelko@tin> | 
  Finished deploy [changeprop/deploy@e4547cd]: Support regexed topics (duration: 00m 55s) | 
  [production] | 
            
  | 17:55 | 
  <ppchelko@tin> | 
  Started deploy [changeprop/deploy@e4547cd]: Support regexed topics | 
  [production] | 
            
  | 17:31 | 
  <godog> | 
  remove ge-3/0/27 from interface-range labs-instance-ports (now for ms-be1031) | 
  [production] | 
            
  | 17:25 | 
  <bblack> | 
  puppet disabled on all cp* ahead of careful deploy for https://gerrit.wikimedia.org/r/#/c/339667/ | 
  [production] | 
            
  | 17:12 | 
  <mutante> | 
  removing parsoid-tests.wikimedia.org from DNS - replaced by more specific parsoid-rt-tests and parsoid-vd-tests | 
  [production] | 
            
  | 17:12 | 
  <nuria@tin> | 
  Finished deploy [eventlogging/analytics@2874077]: (no justification provided) (duration: 00m 03s) | 
  [production] | 
            
  | 17:12 | 
  <nuria@tin> | 
  Started deploy [eventlogging/analytics@2874077]: (no justification provided) | 
  [production] | 
            
  | 17:11 | 
  <elukey> | 
  restarting nginx on eqiad appservers to pick up the new certs | 
  [production] | 
            
  | 16:55 | 
  <marostegui> | 
  Stop eventlog syncs to db1047 and dbstore1002 for maintenance - T160454 | 
  [production] | 
            
  | 16:53 | 
  <marostegui> | 
  Disable puppet on db1047 and dbstore1002 for maintenance - T160454 | 
  [production] | 
            
  | 16:51 | 
  <elukey> | 
  upgrading ssl cert appservers.svc.eqiad.wmnet to include the new discovery endpoints | 
  [production] | 
            
  | 16:51 | 
  <_joe_> | 
  actually performing the parsoid rolling restart in codfw | 
  [production] | 
            
  | 16:31 | 
  <_joe_> | 
  rolling restart of parsoid in codfw | 
  [production] | 
            
  | 14:32 | 
  <moritzm> | 
  installing apparmor security updates on trusty | 
  [production] | 
            
  | 14:31 | 
  <elukey> | 
  upgrading ssl cert api.svc.eqiad.wmnet to include the new discovery endpoints | 
  [production] | 
            
  | 14:14 | 
  <andrewbogott> | 
  disabling puppet on labs hosts for a staged rollout of https://gerrit.wikimedia.org/r/#/c/345275/ | 
  [production] |