| 2016-11-21
      
      § | 
    
  | 11:32 | <gehel> | starting cluster restart on elasticsearch eqiad for JVM upgrade | [production] | 
            
  | 10:29 | <jynus> | performing blocking schema change on db2065 T151029 | [production] | 
            
  | 10:11 | <UltimateKarate> | blah blah blah | [production] | 
            
  | 10:11 | <marostegui> | Deploy ALTER table db1068 commonswiki.revision - https://phabricator.wikimedia.org/T147305 | [production] | 
            
  | 10:05 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1068 - T147305 (duration: 00m 48s) | [production] | 
            
  | 09:49 | <ema> | removed varnishrls.service from non-cache_text hosts | [production] | 
            
  | 08:50 | <elukey> | rolling restart of hadoop-related java daemons on analytics* hosts due to openjdk update | [production] | 
            
  | 08:08 | <marostegui> | Deploy ALTER table db1069 commonswiki.revision - https://phabricator.wikimedia.org/T147305 | [production] | 
            
  | 07:18 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Repool db2070 - T149553 (duration: 00m 49s) | [production] | 
            
  | 06:11 | <legoktm@tin> | Synchronized wmf-config/: Disable centralauth-rename right for maintenance (T148242, T151155) (duration: 00m 52s) | [production] | 
            
  | 02:21 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Mon Nov 21 02:21:35 UTC 2016 (duration 4m 19s) | [production] | 
            
  | 02:17 | <l10nupdate@tin> | scap sync-l10n completed (1.29.0-wmf.3) (duration: 05m 46s) | [production] | 
            
  
    | 2016-11-18
      
      § | 
    
  | 21:32 | <hashar> | CI / Zuul slightly overloaded. Will resolve by itself soon. | [production] | 
            
  | 18:52 | <demon@tin> | Synchronized wmf-config/CommonSettings.php: extdist settings for 1.28 (duration: 00m 49s) | [production] | 
            
  | 18:27 | <jynus> | deployed unix_auth on silver (labswiki) T150446 | [production] | 
            
  | 18:22 | <jynus> | removing mysql-test dir from silver to free up some space there | [production] | 
            
  | 17:34 | <ori@tin> | Synchronized wmf-config/InitialiseSettings.php: If3b80b1a: Revert "Don't use AbuseFilterCachingParser on bgwiki" (T148660) (duration: 00m 50s) | [production] | 
            
  | 16:48 | <ejegg> | disabled job 'Dedupe CiviCRM contacts (name-match)' | [production] | 
            
  | 16:46 | <ejegg> | disabled job 'Dedupe CiviCRM contacts' | [production] | 
            
  | 14:19 | <twentyafterfour> | phabricator: deploying more fixes from upstream/stable to wmf/stable. Fixes T150992 | [production] | 
            
  | 10:13 | <jynus> | performing schema change on dbstore2001:commonswiki/page (ALGORITHM=COPY) | [production] | 
            
  | 09:14 | <marostegui> | Stopping MySQL on db2070 to use it to clone another host - https://phabricator.wikimedia.org/T149553 | [production] | 
            
  | 09:05 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Depool db2070 - T149553 (duration: 00m 48s) | [production] | 
            
  | 08:33 | <elukey> | kafka1022 up and running with kafka* daemon masked and broken disk removed from fstab (we mount partitions in there using UUIDs) | [production] | 
            
  | 08:21 | <marostegui@tin> | Synchronized wmf-config/db-codfw.php: Depool db2049 - T150876 (duration: 00m 49s) | [production] | 
            
  | 08:07 | <_joe_> | physical powercycle of kafka1022 (broken disk) | [production] | 
            
  | 07:55 | <_joe_> | rebooting kafka1022, a shower of defunct processes, kafka refuses to startup again | [production] | 
            
  | 07:43 | <_joe_> | restarting kafka on kafka1022, too many open files | [production] | 
            
  | 07:24 | <moritzm> | installing openjdk-7 security updates on trusty systems | [production] | 
            
  | 02:31 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Fri Nov 18 02:31:52 UTC 2016 (duration 4m 40s) | [production] | 
            
  | 02:27 | <l10nupdate@tin> | scap sync-l10n completed (1.29.0-wmf.3) (duration: 09m 18s) | [production] | 
            
  | 00:11 | <reedy@tin> | Synchronized wmf-config/CommonSettings.php: wfLoadExtension for numerous extensions (duration: 00m 48s) | [production] | 
            
  | 00:10 | <reedy@tin> | Synchronized wmf-config/extension-list: More extensions to extension.json (duration: 00m 48s) | [production] | 
            
  
    | 2016-11-17
      
      § | 
    
  | 23:42 | <krinkle@tin> | Synchronized php-1.29.0-wmf.3/resources/src/mediawiki/mediawiki.js: Ie21f5c: undo temp revert for metric observation (duration: 00m 54s) | [production] | 
            
  | 23:22 | <krinkle@tin> | Synchronized php-1.29.0-wmf.3/resources/src/mediawiki/mediawiki.js: Ie21f5c: temp revert cache-eval for metric observation (duration: 00m 49s) | [production] | 
            
  | 22:24 | <ejegg> | updated SmashPig from 1e895ba4468ac5a2e0812dd5719fd42eb2ec6515 to 3cbb42fb7138e780bd76a2f17c48983f8f882c57 | [production] | 
            
  | 22:08 | <mutante> | contint1001 - Apache fix puppetized, re-enabeld | [production] | 
            
  | 22:02 | <ejegg|food> | updated SmashPig from cac1a196b0e4805ff38671364228a89f8d053c5c to 1e895ba4468ac5a2e0812dd5719fd42eb2ec6515 | [production] | 
            
  | 21:58 | <mutante> | doc.wm.org Apache config live-hack fixed, puppet patch coming up | [production] | 
            
  | 21:02 | <Pchelolo> | update RESTBase to b9722ba7c | [production] | 
            
  | 20:59 | <Pchelolo> | update RESTBase to b9722ba7c - canary on restbase1007 | [production] | 
            
  | 20:55 | <Pchelolo> | update RESTBase to b9722ba7c - staging | [production] | 
            
  | 20:26 | <jynus> | applying schema change on s1 (page) T69223 | [production] | 
            
  | 20:07 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.3 | [production] | 
            
  | 20:03 | <krinkle@tin> | Synchronized php-1.29.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.visibilitychange.js: Ibd0935bef8f (duration: 00m 48s) | [production] |