| 2017-04-27
      
      ยง | 
    
  | 20:53 | <ppchelko@naos> | Finished deploy [restbase/deploy@fcfc537]: Automatically rerender parsoid, only store summaries if they are changed (duration: 11m 33s) | [production] | 
            
  | 20:53 | <twentyafterfour@naos> | rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.21 | [production] | 
            
  | 20:47 | <twentyafterfour@naos> | Synchronized php-1.29.0-wmf.21/extensions/FlaggedRevs: deploy fix for T163994 (duration: 01m 17s) | [production] | 
            
  | 20:42 | <ppchelko@naos> | Started deploy [restbase/deploy@fcfc537]: Automatically rerender parsoid, only store summaries if they are changed | [production] | 
            
  | 20:37 | <mutante> | ocg1001 - has been reinstalled but ocg package deployment fails currently "has the minion key been accepted", should not be repooled just yet | [production] | 
            
  | 20:32 | <mutante> | ores/cache::misc: switch ores back to codfw-only - everything is like it was before the failed deploy yesterday again | [production] | 
            
  | 20:21 | <andrewbogott> | stripping a bunch of unneeded extensions from wikitech-static | [production] | 
            
  | 20:20 | <mutante> | ocg1001 - re-added to puppet, initial run, reinstall ongoing (T161158) | [production] | 
            
  | 20:18 | <mutante> | ores is active/active now, for a short time | [production] | 
            
  | 20:16 | <mutante> | ocg1001 - revoke old puppet cert, salt key | [production] | 
            
  | 20:15 | <mutante> | run puppet on cache::misc to push ores change - cumin -b 5 -s 10 'R:class = role::cache::misc' 'run-puppet-agent -q' | [production] | 
            
  | 20:03 | <twentyafterfour> | 1.29.0-wmf.21 is blocked by T163994 | [production] | 
            
  | 20:01 | <mutante> | ocg1001 - reboot into PXE, re-install | [production] | 
            
  | 19:59 | <twentyafterfour@naos> | Synchronized php-1.29.0-wmf.21/extensions/FlaggedRevs/frontend/FlaggedRevsUI.hooks.php: deploy fix for T163994 (duration: 01m 04s) | [production] | 
            
  | 19:33 | <twentyafterfour> | start mediawiki deployment train group 2 - all wikis to 1.29.0-wmf.21 | [production] | 
            
  | 19:24 | <reedy@naos> | Synchronized wmf-config/CommonSettings.php: Run pdf processors in firejails T164000 (duration: 01m 20s) | [production] | 
            
  | 19:20 | <XenoRyet> | Updated paymentswiki from ee7d402216f6eae71cc85137741187ab096ead33 to 1620b8233321099262ff4333a2269f0563107e66 | [production] | 
            
  | 18:47 | <addshore> | Morning SWAT Done! | [production] | 
            
  | 18:46 | <addshore@naos> | Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:347818|WMDE Spring campaign - Remove logging (no longer needed)]] (duration: 00m 47s) | [production] | 
            
  | 18:44 | <addshore@naos> | Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:347820|wmgUseGettingStarted true for dewiki]] (duration: 00m 48s) | [production] | 
            
  | 18:41 | <addshore@naos> | Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:350140|Enable Cognate Logging]] (duration: 00m 48s) | [production] | 
            
  | 18:40 | <XenoRyet> | Roll back paymentswiki from 030b2f90b155aea8fd3aea4aecf73d0a41d6482b to ee7d402216f6eae71cc85137741187ab096ead33 | [production] | 
            
  | 18:34 | <addshore@naos> | Synchronized php-1.29.0-wmf.21/extensions/CirrusSearch: SWAT [[gerrit:350614|#1]] [[gerrit:350615|#2]] (duration: 00m 59s) | [production] | 
            
  | 18:31 | <addshore@naos> | Synchronized wmf-config/CirrusSearch-common.php: SWAT [[gerrit:350626|update name of sistersearch profile for wikivoyage]] (duration: 00m 49s) | [production] | 
            
  | 18:24 | <addshore@naos> | Synchronized php-1.29.0-wmf.21/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT [[gerrit:350544|WMDE Spring campaign - Remove hook]] PT2/2 (duration: 00m 52s) | [production] | 
            
  | 18:23 | <urandom> | T163936: restarting cassandra-metrics-collector, restbase production | [production] | 
            
  | 18:22 | <addshore@naos> | Synchronized php-1.29.0-wmf.21/extensions/WikimediaEvents/extension.json: SWAT [[gerrit:350544|WMDE Spring campaign - Remove hook]] PT1/2 (duration: 00m 57s) | [production] | 
            
  | 18:21 | <urandom> | T163936: restarting cassandra-metrics-collector, restbase staging | [production] | 
            
  | 18:20 | <addshore@naos> | Synchronized php-1.29.0-wmf.21/includes/api/ApiQueryPagePropNames.php: SWAT [[gerrit:350585|Do not add limit to ApiQueryPagePropNames when database type is mysql]] (duration: 01m 04s) | [production] | 
            
  | 18:17 | <twentyafterfour> | restarting apache on iridium to hotfix T164005 | [production] | 
            
  | 18:07 | <addshore@naos> | Synchronized wmf-config/Wikibase-production.php: SWAT [[gerrit:350584|Fix echoIcon for wikibase in testwikis]] (duration: 01m 27s) | [production] | 
            
  | 17:44 | <XenoRyet> | Updated paymentswiki  from ee7d402216f6eae71cc85137741187ab096ead33 to 030b2f90b155aea8fd3aea4aecf73d0a41d6482b | [production] | 
            
  | 17:36 | <ladsgroup@naos> | Finished deploy [ores/deploy@68cca85]: (no justification provided) (duration: 21m 50s) | [production] | 
            
  | 17:30 | <_joe_> | started pybal on lvs1006 after network was fixed | [production] | 
            
  | 17:25 | <XenoRyet> | reverted paymentswiki  from 030b2f90b155aea8fd3aea4aecf73d0a41d6482b to ee7d402216f6eae71cc85137741187ab096ead33 | [production] | 
            
  | 17:20 | <XenoRyet> | Updated paymentswiki from ee7d402216f6eae71cc85137741187ab096ead33 to 030b2f90b155aea8fd3aea4aecf73d0a41d6482b | [production] | 
            
  | 17:15 | <ladsgroup@naos> | Started deploy [ores/deploy@68cca85]: (no justification provided) | [production] | 
            
  | 17:15 | <Amir1> | ladsgroup@naos:/srv/deployment/ores/deploy$ scap deploy (T163950) | [production] | 
            
  | 17:12 | <demon@naos> | Pruned MediaWiki: 1.29.0-wmf.18 [keeping static files] (duration: 00m 20s) | [production] | 
            
  | 17:08 | <_joe_> | stop pybal on lvs1006 to stop announcing via BGP | [production] | 
            
  | 17:08 | <demon@naos> | Pruned MediaWiki: 1.29.0-wmf.16 (duration: 00m 13s) | [production] | 
            
  | 17:04 | <demon@naos> | Synchronized scap/plugins/clean.py: One last fix (duration: 01m 04s) | [production] | 
            
  | 16:53 | <gehel> | unbanning all elasticsearch servers in eqiad row D - T148506 | [production] | 
            
  | 16:48 | <demon@naos> | Synchronized scap/plugins/clean.py: --keep-static is nice now. Also need a co-master sync (duration: 01m 28s) | [production] | 
            
  | 16:45 | <andrewbogott> | re-enabling labs instance creation/deletion | [production] | 
            
  | 16:42 | <demon@naos> | Pruned MediaWiki: 1.29.0-wmf.19 [keeping static files] (duration: 00m 15s) | [production] | 
            
  | 16:32 | <gehel> | unbanning elasticsearch servers in eqiad row D - elastic10(17|18|19|20) - T148506 | [production] | 
            
  | 15:56 | <elukey> | restart of jmxtrans on all the hadoop worker nodes | [production] | 
            
  | 15:51 | <andrewbogott> | disabling labs instance create/delete to avoid hilarity during network maintenance | [production] | 
            
  | 15:50 | <elukey> | forced 'service ferm start' on the failed analytics hosts | [production] |