| 
      
        2017-01-09
      
      §
     | 
  
    
  | 21:50 | 
  <akosiaris> | 
  service restart zotero on sca1003, sca1004. Zotero OOMed again as usual | 
  [production] | 
            
  | 19:49 | 
  <robh> | 
  updating librenms.wikimedia.org cert, netmon1001 only system affected | 
  [production] | 
            
  | 18:16 | 
  <paravoid> | 
  rebooting and powercycling mira, CPU frequency throttled, suspecting firmware bug | 
  [production] | 
            
  | 17:51 | 
  <hoo> | 
  Updated the Wikidata property suggester with data from last Monday's JSON dump and applied the T132839 workarounds | 
  [production] | 
            
  | 15:35 | 
  <zeljkof> | 
  finished EU SWAT! | 
  [production] | 
            
  | 15:33 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:331229|Add en.wikinews and es.wikinews as import source in testwiki (T154879)]] (duration: 02m 38s) | 
  [production] | 
            
  | 15:26 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:330983|Enable import from cswiki to arbcom_cswiki (T154799)]] (duration: 02m 38s) | 
  [production] | 
            
  | 15:10 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:328719|Add DW alias for NS_PROJECT_TALK in frwiki (T153952)]] (duration: 02m 36s) | 
  [production] | 
            
  | 15:04 | 
  <zeljkof> | 
  extending EU SWAT, tree more patches left to deploy | 
  [production] | 
            
  | 15:00 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/throttle.php: SWAT: [[gerrit:330855|[throttle] Lift for 2017-01-10/12 + minor cleanup (T154312)]] (duration: 02m 36s) | 
  [production] | 
            
  | 14:52 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:330847|Enable Extension:Babel s category on cswikiversity (T67211)]] (duration: 02m 36s) | 
  [production] | 
            
  | 14:39 | 
  <zfilipin@tin> | 
  Synchronized static/images/project-logos: SWAT: [[gerrit:330719|Add HD logos for multiple projects (T150618)]] (duration: 02m 36s) | 
  [production] | 
            
  | 14:25 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:330685|Add digitalmedia.fws.gov to the whitelist (T154671)]] (duration: 02m 38s) | 
  [production] | 
            
  | 10:38 | 
  <akosiaris> | 
  restart nginx and rcstream on rcs1001.eqiad.wmnet to debug issue with prematurely closed connections and 502 returned to clients. No change witnessed. | 
  [production] | 
            
  | 07:12 | 
  <ebernhardson> | 
  restart elasticsearch on relforge100[12] to adjust ltr logging settings | 
  [production] | 
            
  | 02:45 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Mon Jan  9 02:45:57 UTC 2017 (duration 4m 36s) | 
  [production] | 
            
  | 02:41 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.29.0-wmf.7) (duration: 27m 03s) | 
  [production] | 
            
  
    | 
      
        2017-01-06
      
      §
     | 
  
    
  | 23:08 | 
  <godog> | 
  force puppet run on cache_upload in eqiad to switch thumbs back from codfw | 
  [production] | 
            
  | 22:38 | 
  <demon@tin> | 
  Synchronized multiversion: updateBranchPointers consolidation (duration: 00m 56s) | 
  [production] | 
            
  | 20:14 | 
  <demon@tin> | 
  Synchronized w: Dropping old entry point (duration: 00m 41s) | 
  [production] | 
            
  | 19:35 | 
  <demon@tin> | 
  Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources: I32e0b8f81ca2a2e9ffc0c3a379921e12465815f2 (duration: 00m 40s) | 
  [production] | 
            
  | 19:29 | 
  <demon@tin> | 
  Synchronized php-1.29.0-wmf.7/extensions/UploadWizard/resources/mw.UploadWizard.js: I32e0b8f81ca2a2e9ffc0c3a379921e12465815f2 (duration: 00m 59s) | 
  [production] | 
            
  | 19:17 | 
  <ebernhardson> | 
  restarting elasticsearch on relforge100[12] to test new search-ltr plugin | 
  [production] | 
            
  | 19:07 | 
  <ostriches> | 
  gerrit: Started full reindex of all changes, should be background but will be watching | 
  [production] | 
            
  | 18:59 | 
  <mutante> | 
  gerrit restarting for config change 308753 - will be back in seconds | 
  [production] | 
            
  | 18:13 | 
  <mutante> | 
  mw1205 - restarted hhvm | 
  [production] | 
            
  | 18:08 | 
  <godog> | 
  force puppet run on eqiad cache_upload to switch thumbs to codfw | 
  [production] | 
            
  | 16:17 | 
  <godog> | 
  bounce swift-proxy on ms-fe100[123] leave ms-fe1004 for investigation | 
  [production] | 
            
  | 16:14 | 
  <hashar> | 
  Restarting Nodepool | 
  [production] | 
            
  | 16:05 | 
  <ema> | 
  wiping codfw caches T154758 | 
  [production] | 
            
  | 15:29 | 
  <cmjohnson1> | 
  powering off mw1239 to reseat DIMM | 
  [production] | 
            
  | 15:22 | 
  <papaul> | 
  elastic2025-elastic2036 - signing puppet certs, salt-key, initial run | 
  [production] | 
            
  | 14:53 | 
  <mark> | 
  papaul powercycled asw-a7-codfw 14:50 | 
  [production] | 
            
  | 14:16 | 
  <reedy@tin> | 
  Finished scap: Rebuild message cache for Echo api messages being missing T154110 (duration: 25m 00s) | 
  [production] | 
            
  | 13:51 | 
  <reedy@tin> | 
  Started scap: Rebuild message cache for Echo api messages being missing T154110 | 
  [production] | 
            
  | 10:00 | 
  <ariel@tin> | 
  Synchronized wmf-config/throttle.php: test, noop (duration: 02m 45s) | 
  [production] | 
            
  | 09:24 | 
  <paravoid> | 
  asw-a7-codfw is down, serial console unresponsive | 
  [production] | 
            
  | 09:12 | 
  <ariel@tin> | 
  Synchronized wmf-config/throttle.php: Adjust throttle rule for Maharashtra 'Edit Wikipedia' workshop (VNGIASS) (duration: 02m 46s) | 
  [production] | 
            
  | 07:08 | 
  <moritzm> | 
  installing crypto++ security updates on trusty hosts | 
  [production] |