| 2016-11-18
      
      § | 
    
  | 21:32 | <hashar> | CI / Zuul slightly overloaded. Will resolve by itself soon. | [production] | 
            
  | 18:52 | <demon@tin> | Synchronized wmf-config/CommonSettings.php: extdist settings for 1.28 (duration: 00m 49s) | [production] | 
            
  | 18:27 | <jynus> | deployed unix_auth on silver (labswiki) T150446 | [production] | 
            
  | 18:22 | <jynus> | removing mysql-test dir from silver to free up some space there | [production] | 
            
  | 17:34 | <ori@tin> | Synchronized wmf-config/InitialiseSettings.php: If3b80b1a: Revert "Don't use AbuseFilterCachingParser on bgwiki" (T148660) (duration: 00m 50s) | [production] | 
            
  | 16:48 | <ejegg> | disabled job 'Dedupe CiviCRM contacts (name-match)' | [production] | 
            
  | 16:46 | <ejegg> | disabled job 'Dedupe CiviCRM contacts' | [production] | 
            
  | 14:19 | <twentyafterfour> | phabricator: deploying more fixes from upstream/stable to wmf/stable. Fixes T150992 | [production] | 
            
  | 10:13 | <jynus> | performing schema change on dbstore2001:commonswiki/page (ALGORITHM=COPY) | [production] | 
            
  | 09:14 | <marostegui> | Stopping MySQL on db2070 to use it to clone another host - https://phabricator.wikimedia.org/T149553 | [production] | 
            
  | 09:05 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Depool db2070 - T149553 (duration: 00m 48s) | [production] | 
            
  | 08:33 | <elukey> | kafka1022 up and running with kafka* daemon masked and broken disk removed from fstab (we mount partitions in there using UUIDs) | [production] | 
            
  | 08:21 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Depool db2049 - T150876 (duration: 00m 49s) | [production] | 
            
  | 08:07 | <_joe_> | physical powercycle of kafka1022 (broken disk) | [production] | 
            
  | 07:55 | <_joe_> | rebooting kafka1022, a shower of defunct processes, kafka refuses to startup again | [production] | 
            
  | 07:43 | <_joe_> | restarting kafka on kafka1022, too many open files | [production] | 
            
  | 07:24 | <moritzm> | installing openjdk-7 security updates on trusty systems | [production] | 
            
  | 02:31 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Fri Nov 18 02:31:52 UTC 2016 (duration 4m 40s) | [production] | 
            
  | 02:27 | <l10nupdate@tin> | scap sync-l10n completed (1.29.0-wmf.3) (duration: 09m 18s) | [production] | 
            
  | 00:11 | <reedy@tin> | Synchronized wmf-config/CommonSettings.php: wfLoadExtension for numerous extensions (duration: 00m 48s) | [production] | 
            
  | 00:10 | <reedy@tin> | Synchronized wmf-config/extension-list: More extensions to extension.json (duration: 00m 48s) | [production] | 
            
  
    | 2016-11-17
      
      § | 
    
  | 23:42 | <krinkle@tin> | Synchronized php-1.29.0-wmf.3/resources/src/mediawiki/mediawiki.js: Ie21f5c: undo temp revert for metric observation (duration: 00m 54s) | [production] | 
            
  | 23:22 | <krinkle@tin> | Synchronized php-1.29.0-wmf.3/resources/src/mediawiki/mediawiki.js: Ie21f5c: temp revert cache-eval for metric observation (duration: 00m 49s) | [production] | 
            
  | 22:24 | <ejegg> | updated SmashPig from 1e895ba4468ac5a2e0812dd5719fd42eb2ec6515 to 3cbb42fb7138e780bd76a2f17c48983f8f882c57 | [production] | 
            
  | 22:08 | <mutante> | contint1001 - Apache fix puppetized, re-enabeld | [production] | 
            
  | 22:02 | <ejegg|food> | updated SmashPig from cac1a196b0e4805ff38671364228a89f8d053c5c to 1e895ba4468ac5a2e0812dd5719fd42eb2ec6515 | [production] | 
            
  | 21:58 | <mutante> | doc.wm.org Apache config live-hack fixed, puppet patch coming up | [production] | 
            
  | 21:02 | <Pchelolo> | update RESTBase to b9722ba7c | [production] | 
            
  | 20:59 | <Pchelolo> | update RESTBase to b9722ba7c - canary on restbase1007 | [production] | 
            
  | 20:55 | <Pchelolo> | update RESTBase to b9722ba7c - staging | [production] | 
            
  | 20:26 | <jynus> | applying schema change on s1 (page) T69223 | [production] | 
            
  | 20:07 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.3 | [production] | 
            
  | 20:03 | <krinkle@tin> | Synchronized php-1.29.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.visibilitychange.js: Ibd0935bef8f (duration: 00m 48s) | [production] | 
            
  | 19:58 | <ejegg> | extended session timeouts on payments-wiki | [production] | 
            
  | 19:25 | <addshore@tin> | Synchronized php-1.29.0-wmf.3/extensions/Wikidata: {{gerrit|322092}} T150948 Backporting fix for quantity precision issue. (duration: 02m 17s) | [production] | 
            
  | 19:07 | <krenair@tin> | Synchronized wmf-config/interwiki.php: https://gerrit.wikimedia.org/r/#/c/322141/ (duration: 00m 50s) | [production] | 
            
  | 18:57 | <bearND> | deployed mobileapps bf44547 | [production] | 
            
  | 18:53 | <bearND> | starting mobileapps deploy | [production] | 
            
  | 18:36 | <twentyafterfour> | phabricator: deploy upstream fix for T150971 (upstream sha1: 7ebc47d906fe ) | [production] | 
            
  | 18:16 | <twentyafterfour> | deploy more phabricator hotfixes | [production] | 
            
  | 17:46 | <twentyafterfour> | unbreak search | [production] | 
            
  | 17:40 | <twentyafterfour> | phabricator: deploying hotfix for T150965 | [production] | 
            
  | 17:34 | <godog> | bounce gerrit on cobalt after https://gerrit.wikimedia.org/r/321398 | [production] | 
            
  | 17:11 | <twentyafterfour> | restart apache2 on iridium to clear lagged queries refs T150965 | [production] | 
            
  | 17:07 | <ejegg> | increased paypal job runner message limit to 5000 | [production] | 
            
  | 17:02 | <cmjohnson> | powering down ms-be1016 to reseat the raid card. | [production] | 
            
  | 16:52 | <mobrovac> | change-prop deploying ed3711b | [production] |