| 2015-10-27
      
      § | 
    
  | 23:53 | <tgr@tin> | Started scap: Updating MediaViewer with r246112 | [production] | 
            
  | 23:13 | <ori> | running sync-common on mw2050, mw2128, mw2187 and mw2209 (cf I324134438955c7) | [production] | 
            
  | 23:12 | <gwicke> | ran `sudo mdadm --readwrite md1` on restbase1007 to resolve `pending` state | [production] | 
            
  | 23:11 | <gwicke> | rebooted restbase1007 to rule out a funky hardware state causing elevated read latencies | [production] | 
            
  | 23:05 | <krenair@tin> | Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s) | [production] | 
            
  | 23:05 | <krenair@tin> | Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s) | [production] | 
            
  | 23:05 | <krenair@tin> | Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 18s) | [production] | 
            
  | 23:01 | <legoktm@tin> | Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/249306, no-op (duration: 00m 18s) | [production] | 
            
  | 22:39 | <mutante> | powercycling unresponsive analytics1039, here's what i saw on mgmt https://phabricator.wikimedia.org/P2248 | [production] | 
            
  | 21:50 | <aaron@tin> | Synchronized php-1.27.0-wmf.4/includes/MovePage.php: 7a7c7b27d6c (duration: 00m 17s) | [production] | 
            
  | 21:50 | <catrope@tin> | Synchronized php-1.27.0-wmf.3/extensions/Flow/: Fix for cache key bug (duration: 00m 20s) | [production] | 
            
  | 21:29 | <twentyafterfour@tin> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.4 | [production] | 
            
  | 21:29 | <ori@tin> | Synchronized php-1.27.0-wmf.3/includes/debug/logger/LoggerFactory.php: I437bcb532: LoggerFactory: Only check for Psr\Log\LoggerInterface once (duration: 00m 18s) | [production] | 
            
  | 19:46 | <twentyafterfour@tin> | Finished scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch (duration: 31m 32s) | [production] | 
            
  | 19:44 | <bd808> | Ran sync-common on mw2187 to rebuild l10n caches | [production] | 
            
  | 19:15 | <twentyafterfour@tin> | Started scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch | [production] | 
            
  | 17:56 | <paravoid> | updating jessie debian-installer to 20150422+deb8u2 | [production] | 
            
  | 17:50 | <Jeff_Green> | add fundraising-banner-logger hosts to icinga/nsca | [production] | 
            
  | 17:40 | <krinkle@tin> | Synchronized php-1.27.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js: I95db9deefe363a65 (duration: 00m 17s) | [production] | 
            
  | 17:34 | <ori@tin> | Synchronized php-1.27.0-wmf.3/resources/src/mediawiki.ui: I54c195541: Get rid of CSS transitions on form elements in mediawiki.ui (duration: 00m 17s) | [production] | 
            
  | 16:07 | <Jeff_Green> | round of fundraising OS updates, occasional icinga noise is expected | [production] | 
            
  | 16:02 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.3/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: SWAT: mw.ForeignStructuredUpload: Mark description as being in source wikis content language [[gerrit:249081]] (duration: 00m 17s) | [production] | 
            
  | 15:41 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor in the "Projet" namespace on the French Wikipedia [[gerrit:248910]] (duration: 00m 17s) | [production] | 
            
  | 15:15 | <godog> | reenable puppet on graphite1001 | [production] | 
            
  | 15:01 | <kart_> | TT112626 Ran fix-stats.php for CX (from bewiki to ruwiki) | [production] | 
            
  | 13:00 | <krenair@tin> | Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/249107/ (duration: 00m 18s) | [production] | 
            
  | 12:58 | <akosiaris> | disabling puppet and bringing down OTRS service on mendelevium | [production] | 
            
  | 12:21 | <jynus> | Just dropped msg_resource tables from labs dbs. Filters modified to stop replicationg them. Started replicating the heartbeat tables. | [production] | 
            
  | 11:56 | <godog> | reimage restbase-test2003 | [production] | 
            
  | 11:23 | <godog> | cassandra OOM'd on restbase1007, restarting | [production] | 
            
  | 10:57 | <godog> | downtime restbase endpoints health for restbase1* while investigating | [production] | 
            
  | 10:56 | <hashar> | Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-doc/ is broken. I am on it :-( | [production] | 
            
  | 10:50 | <hashar> | stopping Jenkins due to an unclean state | [production] | 
            
  | 10:11 | <jynus> | disabling puppet and restarting mysql servers at db1069- this will create a small amount of lag on labs | [production] | 
            
  | 09:54 | <godog> | convert restbase-test2003 to cassandra multi-instance | [production] | 
            
  | 09:08 | <aude@tin> | Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add forceParse UpdaterFlag and option in forceSearchIndex script (duration: 00m 19s) | [production] | 
            
  | 05:47 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Tue Oct 27 05:47:02 UTC 2015 (duration 47m 1s) | [production] | 
            
  | 03:42 | <krinkle@tin> | Synchronized php-1.27.0-wmf.3/languages/Language.php: hotfix for T116693 (duration: 00m 19s) | [production] | 
            
  | 02:29 | <l10nupdate@tin> | LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-27 02:29:08+00:00 | [production] | 
            
  | 02:24 | <l10nupdate@tin> | Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 06s) | [production] | 
            
  | 01:02 | <krinkle@tin> | Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: T116693 (duration: 00m 19s) | [production] | 
            
  | 00:42 | <ori@tin> | Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: I2f84cff0: Avoid pointless range scan for 'load-recent-authors' (T116557) (duration: 00m 18s) | [production] | 
            
  | 00:11 | <krenair@tin> | Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Parsoid/Utils.php: https://gerrit.wikimedia.org/r/#/c/249026 (duration: 00m 18s) | [production] |