| 2016-07-14
      
      § | 
    
  | 14:47 | <aude@tin> | scap aborted: (no message) (duration: 00m 02s) | [production] | 
            
  | 14:47 | <aude@tin> | Started scap: (no message) | [production] | 
            
  | 14:45 | <aude@tin> | Synchronized wmf-config/CommonSettings.php: Enable RevisionSlider on test wikis (duration: 00m 28s) | [production] | 
            
  | 14:44 | <aude@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable RevisionSlider on test wikis (duration: 00m 27s) | [production] | 
            
  | 14:38 | <aude@tin> | Synchronized wmf-config/extension-list: Add RevisionSlider to extension-list (duration: 00m 42s) | [production] | 
            
  | 09:58 | <godog> | powercycle ms-be1012, adding back replaced disk | [production] | 
            
  | 08:40 | <Amir1> | for ores | [production] | 
            
  | 08:40 | <Amir1> | deploying 0e9555f to scb nodes | [production] | 
            
  | 08:39 | <elukey> | restarted hhvm on mw1289 mw1280 mw1288 mw1284 mw1287 | [production] | 
            
  | 08:18 | <jynus> | running "megacli -PDOffline -PhysDrv '[32:6]' -aALL" on dbstore1002 to debug issue T140337 | [production] | 
            
  | 08:06 | <elukey> | upgrading cache misc to varnishkafka 1.0.11-1 | [production] | 
            
  | 08:03 | <_joe_> | removing appservers mw1018-25 from service via conftool for decommissioning (T139353) | [production] | 
            
  | 08:01 | <elukey> | removing api servers mw111[4-9] from service via conftool as first decom step (T139353) | [production] | 
            
  | 07:55 | <elukey> | removing api servers mw112[0-9] from service via conftool as first decom step (T139353) | [production] | 
            
  | 06:45 | <moritzm> | restarted hhvm on mw1170 | [production] | 
            
  | 06:07 | <moritzm> | upgrading hhvm in codfw | [production] | 
            
  | 04:22 | <twentyafterfour> | Phabricator hotfix: applied patch to disable task graph on tasks with > 100 related tasks. | [production] | 
            
  | 03:14 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Thu Jul 14 03:14:57 UTC 2016 (duration 7m 16s) | [production] | 
            
  | 03:07 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.10) (duration: 15m 50s) | [production] | 
            
  | 02:37 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.8) (duration: 15m 17s) | [production] | 
            
  | 00:36 | <matt_flaschen> | Ran backfillReadBundles on labtestwiki | [production] | 
            
  | 00:35 | <matt_flaschen> | Started backfillReadBundles on labswiki | [production] | 
            
  | 00:24 | <matt_flaschen> | Started backfillUnreadWikis --rebuild and backfillReadBundles for all group 0 and group 1 wikis earlier | [production] | 
            
  | 00:06 | <twentyafterfour> | Phabricator maintenance completed. Service restored | [production] | 
            
  | 00:01 | <twentyafterfour> | preparing to take Phabricator offline momentarily for scheduled maintenance / upgrade. Service should be restored within a couple of minutes. | [production] | 
            
  
    | 2016-07-13
      
      § | 
    
  | 23:31 | <eileen> | update CiviCRM from 0898bb9360fe4a5ddea1a41d4e3f3e9823afee27 to cea316cc57c511c645a92a003028c95e19cac877 | [production] | 
            
  | 23:27 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/298822 (duration: 00m 26s) | [production] | 
            
  | 23:25 | <krenair@tin> | Synchronized static/images/project-logos: update for https://gerrit.wikimedia.org/r/298819 and https://gerrit.wikimedia.org/r/298822 (duration: 00m 24s) | [production] | 
            
  | 23:17 | <krenair@tin> | Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/225509 + https://gerrit.wikimedia.org/r/298899 - create https://meta.wikimedia.org/wiki/Special:Contact/stewards (duration: 00m 26s) | [production] | 
            
  | 23:17 | <ori@tin> | Synchronized php-1.28.0-wmf.10/includes/resourceloader/ResourceLoaderStartUpModule.php: I882bf7075: ResourceLoader: Update expected length of module version hash (duration: 00m 25s) | [production] | 
            
  | 21:55 | <mutante> | ytterbium - puppet enabled again, fix deployed | [production] | 
            
  | 21:48 | <mutante> | ytterbium, disabled puppet, started apache, needs fix | [production] | 
            
  | 20:40 | <demon@tin> | Synchronized README: no-op to bring co-masters in sync (duration: 00m 28s) | [production] | 
            
  | 20:27 | <bearND> | deployed mobileapps d1eb1da | [production] | 
            
  | 20:20 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10 | [production] | 
            
  | 20:19 | <bearND> | starting mobileapps deploy | [production] | 
            
  | 20:03 | <ebernhard|lunch> | Update codfw elasticsearch cluster sttings with cluster.routing.allocation.disk.watermark.low: 70% to match eqiad and reduce free space icinga warnings | [production] | 
            
  | 19:28 | <ostriches> | gerrit: restarting, puppet back on, issue fixed. | [production] | 
            
  | 19:23 | <anomie@tin> | Synchronized php-1.28.0-wmf.10/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 28s) | [production] | 
            
  | 19:23 | <anomie@tin> | Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 27s) | [production] | 
            
  | 19:14 | <legoktm@tin> | Synchronized php-1.28.0-wmf.8/includes/api/ApiQueryRecentChanges.php: API: Remove index forcing in ApiQueryRecentChanges - T140108 (duration: 00m 26s) | [production] | 
            
  | 19:04 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: all group0 to wmf.10 | [production] | 
            
  | 18:41 | <ostriches> | gerrit/ytterbium: flapped for a minute because of incompat 2.12/2.8 config. Working, puppet disabled pending real fix. | [production] | 
            
  | 18:40 | <mutante> | gerrit has a temp problem. maintenance going on | [production] | 
            
  | 18:34 | <krenair@tin> | Synchronized wmf-config: labs-only change, should be a noop here: https://gerrit.wikimedia.org/r/298812 (duration: 00m 27s) | [production] | 
            
  | 18:32 | <mutante> | gerrit will restart shortly for a config change. expect a very short downtime | [production] | 
            
  | 18:22 | <legoktm> | checkLocalUser.php finished, starting run #2 now | [production] | 
            
  | 17:58 | <legoktm@tin> | Synchronized wmf-config/: PoolCounterClient.php -> extension.json (duration: 00m 32s) | [production] | 
            
  | 17:44 | <legoktm@tin> | Synchronized php-1.28.0-wmf.10/extensions/WikimediaEvents/WikimediaEventsHooks.php: Include the namespace for all pages & Include the resolved special page name for special pages - T138500 (duration: 00m 36s) | [production] | 
            
  | 17:41 | <demon@tin> | Finished scap: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen (once more with feeling) (duration: 47m 06s) | [production] |