| 2016-07-13
      
      § | 
    
  | 13:20 | <gehel> | scheduling icinga downtime on elastic1001-1016 prior to decommissioning (T139758) | [production] | 
            
  | 13:01 | <elukey> | upgrading cache maps to varnishkafka 1.0.11-1 | [production] | 
            
  | 13:00 | <elukey> | uploaded varnishkafka 1.0.11-1 to jessie-wikimedia experimental | [production] | 
            
  | 12:52 | <hashar> | CI is processing with Zuul 2.1.0-151-g30a433b.  It might stop processing events at anytime though due to T137525 | [production] | 
            
  | 12:36 | <hashar> | T137525 Upgrading Zuul 2.1.0-95-g66c8e52-wmf1precise1 ... zuul_2.1.0-151-g30a433b-wmf1precise1_amd64.deb | [production] | 
            
  | 11:52 | <elukey> | installing varnishkafka_1.0.11-1 on cp3008.esams to test it before the complete rollout | [production] | 
            
  | 11:38 | <paravoid> | cr1-ulsfo: "restart snmp" to fix SNMP hiccup after reboot | [production] | 
            
  | 11:38 | <paravoid> | cr1/2-ulsfo: disabling flow monitoring | [production] | 
            
  | 04:22 | <legoktm@tin> | Synchronized wmf-config/InitialiseSettings.php: revert http logging change (duration: 00m 31s) | [production] | 
            
  | 04:20 | <legoktm@tin> | Synchronized wmf-config/interwiki.php: Update interwiki map, make them HTTPS (duration: 00m 39s) | [production] | 
            
  | 03:44 | <legoktm@tin> | Synchronized wmf-config/InitialiseSettings.php: Log 'http' at warning level to debug transwiki import errors (duration: 00m 29s) | [production] | 
            
  | 02:34 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Wed Jul 13 02:34:52 UTC 2016 (duration 6m 8s) | [production] | 
            
  | 02:28 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.8) (duration: 10m 11s) | [production] | 
            
  | 00:56 | <ori> | Restarted grrrit-wm | [production] | 
            
  | 00:41 | <maxsem@tin> | Synchronized php-1.28.0-wmf.8/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#/c/298677/ (duration: 01m 01s) | [production] | 
            
  | 00:07 | <dereckson@tin> | Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: getCentralAuthToken visibility back to protected ([[Gerrit:298661]]) (duration: 00m 27s) | [production] | 
            
  
    | 2016-07-12
      
      § | 
    
  | 23:43 | <dereckson@tin> | Synchronized wmf-config/InitialiseSettings.php: Add flow-create-board for gomwiki sysop (T139226) (duration: 00m 27s) | [production] | 
            
  | 23:41 | <mutante> | lithium deleted some logs older than 60 days to make space | [production] | 
            
  | 23:33 | <dereckson@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable lazy loaded references and images on Thai wikipedia (T136731) (duration: 00m 38s) | [production] | 
            
  | 23:29 | <mutante> | stat1003 still on every puppet run a mongodb gets started..over and over again | [production] | 
            
  | 23:12 | <ejegg> | updated payments from d9f7027340e5311f38c4224c2fddde087467df87 to 8bf6e911eb43a2d369bf656f07d1b51be0a54f6c | [production] | 
            
  | 23:12 | <ostriches> | lead: puppet disabled for a bit while index building is in progress. | [production] | 
            
  | 23:12 | <ostriches> | ytterbium: puppet enabled again, all happy again | [production] | 
            
  | 22:56 | <ostriches> | ytterbium: disabled puppet for a moment so we can do a config change w/o gerrit restarting itself | [production] | 
            
  | 22:06 | <eileen> | upgrade Civicrm from f7434730ebd87f6d542c34c080c61eb3f21ccc6b to 0898bb9360fe4a5ddea1a41d4e3f3e9823afee27 | [production] | 
            
  | 21:52 | <eileen> | Updating CiviCRM from 415d7e62bc3bcbd7c5e3682da64ee4847ad63f5b to f7434730ebd87f6d542c34c080c61eb3f21ccc6b | [production] | 
            
  | 21:24 | <mattflaschen@tin> | Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T140144: Echo/CentralAuth: Bail if not fully initialized (duration: 00m 49s) | [production] | 
            
  | 20:18 | <urandom> | Start revision culling script for local_group_wikipedia_T_parsoid_html, from restbase1009.eqiad.wmnet : T140008 | [production] | 
            
  | 19:30 | <mattflaschen@tin> | Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: T140144: Troubleshoot why Echo is still triggering CA failures (duration: 00m 39s) | [production] | 
            
  | 19:01 | <mutante> | git pulled on strontium to sync with palladium | [production] | 
            
  | 18:48 | <matt_flaschen> | Started checkLocalUser.php at ~2016-07-12 17:45 UTC, killed ~18:06 since Echo apparently is not fully fixed after all. | [production] | 
            
  | 18:38 | <legoktm> | foreachwiki ../../../../home/legoktm/checkLocalUser.php --delete=1 --verbose=1  on terbium | [production] | 
            
  | 18:24 | <anomie@tin> | Synchronized php-1.28.0-wmf.9/includes/auth/AuthManager.php: Commit transaction after auto-creating a user [[gerrit:298541]] (duration: 00m 29s) | [production] | 
            
  | 18:22 | <anomie@tin> | Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Commit transaction after auto-creating a user [[gerrit:298540]] (duration: 00m 30s) | [production] | 
            
  | 18:08 | <demon@tin> | Synchronized wmf-config/InitialiseSettings.php: turn cx back on (duration: 00m 29s) | [production] | 
            
  | 17:44 | <demon@tin> | Synchronized php-1.28.0-wmf.8/extensions/ContentTranslation/: ping limiter fixes (duration: 00m 29s) | [production] | 
            
  | 17:37 | <jynus> | out of band ALTER TABLE recentchanges ADD KEY `name_type_patrolled_timestamp` on db1054 T140108 | [production] | 
            
  | 16:58 | <demon@tin> | Synchronized wmf-config/InitialiseSettings.php: prep pinglimiter config for content translation (duration: 00m 33s) | [production] | 
            
  | 16:56 | <Amir1> | deploying ores f472f65 to scb | [production] | 
            
  | 16:51 | <Amir1> | deploying ores f472f65 to scb2001 | [production] | 
            
  | 16:42 | <godog> | disable puppet on ms-fe* and re-enable gradually to apply https://gerrit.wikimedia.org/r/#/c/298297/ | [production] | 
            
  | 16:25 | <mattflaschen@tin> | Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: ForeignWikiRequest: Bail early for non-global users (duration: 00m 32s) | [production] | 
            
  | 16:12 | <demon@tin> | Synchronized wmf-config/InitialiseSettings.php: Disable content translation, outage right now (duration: 00m 29s) | [production] | 
            
  | 15:44 | <bblack> | cache nodes: salt manual removal of vm compaction cron via sed ( https://gerrit.wikimedia.org/r/298499 ) | [production] | 
            
  | 15:33 | <thcipriani@tin> | Synchronized php-1.28.0-wmf.9/extensions/Echo/includes/ForeignWikiRequest.php: SWAT: [[gerrit:298429|ForeignWikiRequest: Bail early for non-global users (T119736)]] (duration: 00m 31s) | [production] | 
            
  | 15:29 | <thcipriani@tin> | Synchronized portals: SWAT: [[gerrit:298476|Bumping portals to master (T128546)]] (duration: 00m 29s) | [production] | 
            
  | 15:28 | <thcipriani@tin> | Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:298476|Bumping portals to master (T128546)]] (duration: 00m 29s) | [production] | 
            
  | 15:20 | <bblack> | upgrading nginx to 1.11.2-1+wmf1 on all caches | [production] | 
            
  | 15:18 | <thcipriani@tin> | Synchronized static/images/project-logos/trwikimedia.png: SWAT: Revert [[gerrit:298441|Logo update for trwikimedia (T140015)]] (duration: 00m 29s) | [production] | 
            
  | 15:09 | <thcipriani@tin> | Synchronized static/images/project-logos/trwikimedia.png: SWAT: [[gerrit:298441|Logo update for trwikimedia (T140015)]] (duration: 00m 33s) | [production] |