| 2017-03-30
      
      § | 
    
  | 09:48 | <root@tin> | Synchronized wmf-config/db-eqiad.php: Uniform maintenance message and indentation (duration: 00m 47s) | [production] | 
            
  | 09:34 | <root@tin> | Synchronized wmf-config/db-codfw.php: Uniform maintenance message and indentation (duration: 00m 44s) | [production] | 
            
  | 09:06 | <elukey@puppetmaster1001> | conftool action : set/pooled=no; selector: name=mw1261.eqiad.wmnet | [production] | 
            
  | 09:05 | <elukey> | depooling mw1261 (hhvm-dump-debug in /tmp/hhvm.98736.bt.) | [production] | 
            
  | 08:38 | <moritzm> | repooling mw1261 to reproduce hhvm deadlock with higher debug level | [production] | 
            
  | 08:13 | <marostegui> | Convert UNIQUE keys to PK on db1090 (s2) - T17441 | [production] | 
            
  | 08:03 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1090 - T17441 (duration: 00m 44s) | [production] | 
            
  | 07:43 | <gehel@puppetmaster1001> | conftool action : set/pooled=yes; selector: name=elastic2021.codfw.wmnet | [production] | 
            
  | 07:41 | <gehel> | pull elastic2021 back into active duty - T149006 | [production] | 
            
  | 07:05 | <ema> | upgrading twisted to 16.2.0 on lvs100[123] (eqiad primaries) T160433 | [production] | 
            
  | 06:45 | <moritzm> | installing apparmor security updates on trusty | [production] | 
            
  | 06:25 | <marostegui> | Logging backwards for the record: restart mysql on db1047 for maintenance - T160454 | [production] | 
            
  | 05:56 | <marostegui> | Deploy schema change on db2014 - codfw master (this will generate lag on codfw) - T73563 | [production] | 
            
  | 05:56 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1094 - T17441 (duration: 00m 45s) | [production] | 
            
  | 05:51 | <ema> | upgrading twisted to 16.2.0 on lvs100[456] (eqiad secondaries) T160433 | [production] | 
            
  | 03:03 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Thu Mar 30 03:03:31 UTC 2017 (duration 5m 49s) | [production] | 
            
  | 02:57 | <l10nupdate@tin> | scap sync-l10n completed (1.29.0-wmf.18) (duration: 13m 43s) | [production] | 
            
  | 02:22 | <l10nupdate@tin> | scap sync-l10n completed (1.29.0-wmf.17) (duration: 07m 20s) | [production] | 
            
  | 01:52 | <twentyafterfour> | phd fixed on iridium. libphutil was out of sync with phd source | [production] | 
            
  | 01:11 | <twentyafterfour> | running `puppet agent --test` on iridium | [production] | 
            
  | 00:10 | <twentyafterfour> | Phabricator update completed. | [production] | 
            
  | 00:10 | <mutante> | ruthenium low on disk space, because /srv/visualdiff/pngs (parsoid-vd-tests) is pretty large and /srv isn't a separate mount | [production] | 
            
  | 00:06 | <twentyafterfour> | updating phabricator on iridium | [production] | 
            
  | 00:04 | <mutante> | ruthenium - apt-get clean gets a little more disk space | [production] | 
            
  
    | 2017-03-29
      
      § | 
    
  | 23:46 | <reedy@tin> | Synchronized php-1.29.0-wmf.18/extensions/Quiz: Fix undefined variable stateObject T161735 (duration: 00m 49s) | [production] | 
            
  | 23:43 | <reedy@tin> | Synchronized wmf-config/CommonSettings.php: Dont use EP_NS in CommonSettings (duration: 00m 44s) | [production] | 
            
  | 21:23 | <krinkle@tin> | Synchronized errorpages/: I15295835a1a (duration: 00m 44s) | [production] | 
            
  | 20:56 | <thcipriani@tin> | Synchronized php-1.29.0-wmf.18/extensions/ProofreadPage/includes/page/ProofreadPagePage.php: [[gerrit:345423|Makes sure to always return a Title in ProofreadPagePage::findIndexTitle]] T161734 (duration: 00m 46s) | [production] | 
            
  | 20:35 | <halfak@tin> | Finished deploy [ores/deploy@554ea12]: T160638 (duration: 18m 40s) | [production] | 
            
  | 20:31 | <arlolra> | Updated Parsoid to b1b27146 (T161558, T160207, T153798) | [production] | 
            
  | 20:21 | <arlolra@tin> | Finished deploy [parsoid/deploy@bc798dc]: Updating Parsoid to b1b27146 (duration: 07m 26s) | [production] | 
            
  | 20:16 | <halfak@tin> | Started deploy [ores/deploy@554ea12]: T160638 | [production] | 
            
  | 20:13 | <arlolra@tin> | Started deploy [parsoid/deploy@bc798dc]: Updating Parsoid to b1b27146 | [production] | 
            
  | 20:09 | <ppchelko@tin> | Finished deploy [changeprop/deploy@ef62908]: Fix metrics for regex topics (duration: 00m 56s) | [production] | 
            
  | 20:08 | <ppchelko@tin> | Started deploy [changeprop/deploy@ef62908]: Fix metrics for regex topics | [production] | 
            
  | 19:46 | <ppchelko@tin> | Finished deploy [changeprop/deploy@1150cf5]: Config: Enabling regex-based topic subscription (duration: 01m 45s) | [production] | 
            
  | 19:44 | <ppchelko@tin> | Started deploy [changeprop/deploy@1150cf5]: Config: Enabling regex-based topic subscription | [production] | 
            
  | 19:16 | <awight> | re-run today's ingenico audit job | [production] | 
            
  | 19:15 | <awight> | pick at paypal scab: re-run audit parser | [production] | 
            
  | 19:10 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.29.0-wmf.17 | [production] | 
            
  | 19:05 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.18 | [production] | 
            
  | 18:45 | <bblack> | varnish active/active deploy done ( https://gerrit.wikimedia.org/r/#/c/339667/ ) - all caches running the new code, puppet re-enabled, etc. | [production] | 
            
  | 18:43 | <hoo> | Started a Wikidata TTL dump run on snapshot1007 using Zend (due to T161695). | [production] | 
            
  | 18:22 | <catrope@tin> | Synchronized php-1.29.0-wmf.18/includes/page/WikiPage.php: T159319 (duration: 00m 44s) | [production] | 
            
  | 18:22 | <catrope@tin> | Synchronized php-1.29.0-wmf.18/includes/Title.php: T159319 (duration: 00m 46s) | [production] | 
            
  | 18:10 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Save->Publihs on Wikipedias except dewiki and enwiki (T131132); set wgOOUIEditPage false everywhere (duration: 00m 57s) | [production] | 
            
  | 17:58 | <awight> | disabling PayPal audit parser | [production] | 
            
  | 17:56 | <ppchelko@tin> | Finished deploy [changeprop/deploy@e4547cd]: Support regexed topics (duration: 00m 55s) | [production] | 
            
  | 17:55 | <ppchelko@tin> | Started deploy [changeprop/deploy@e4547cd]: Support regexed topics | [production] | 
            
  | 17:31 | <godog> | remove ge-3/0/27 from interface-range labs-instance-ports (now for ms-be1031) | [production] |