| 
      
        2016-03-29
      
      §
     | 
  
    
  | 15:29 | 
  <thcipriani@tin> | 
  Synchronized wmf-config: SWAT: Use extension registration for SyntaxHighlight_GeSHi [[gerrit:280224]] (duration: 00m 28s) | 
  [production] | 
            
  | 15:26 | 
  <thcipriani@tin> | 
  Synchronized wmf-config: SWAT: Use extension registration for ImageMap [[gerrit:280223]] (duration: 00m 31s) | 
  [production] | 
            
  | 15:12 | 
  <thcipriani@tin> | 
  Synchronized static/images/project-logos/astwiktionary.png: SWAT: Logo for ast.wiktionary [[gerrit:280057]] (duration: 00m 27s) | 
  [production] | 
            
  | 15:08 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: HD logo for da.wikipedia [[gerrit:279897]] (duration: 00m 27s) | 
  [production] | 
            
  | 15:07 | 
  <thcipriani@tin> | 
  Synchronized static/images/project-logos: SWAT: HD logo for da.wikipedia [[gerrit:279897]] (duration: 00m 38s) | 
  [production] | 
            
  | 15:06 | 
  <moritzm> | 
  repooled maps-test2004 | 
  [production] | 
            
  | 15:02 | 
  <moritzm> | 
  repooled maps-test2003 and depooled maps-test2004 (to apply ferm) | 
  [production] | 
            
  | 14:54 | 
  <moritzm> | 
  depooled maps-test2003 (to apply ferm) | 
  [production] | 
            
  | 13:53 | 
  <moritzm> | 
  repooled maps-test2002 (nodejs being put on hold prevented generation of ferm rules provided by service::node, this has been fixed) | 
  [production] | 
            
  | 13:14 | 
  <jynus> | 
  restoring (but not provisioning) older backup from labswiki sql | 
  [production] | 
            
  | 13:08 | 
  <moritzm> | 
  unheld nodejs on maps* | 
  [production] | 
            
  | 11:51 | 
  <hashar> | 
  Jenkins / Zuul lagging out trying to catch up with a huge number of changes I have sent | 
  [production] | 
            
  | 10:11 | 
  <reedy@tin> | 
  Synchronized wmf-config/throttle.php: Remove old throttle rules (duration: 00m 44s) | 
  [production] | 
            
  | 09:50 | 
  <godog> | 
  powercycle ms-be2008, got stuck again while diagnosing failed /dev/sdl | 
  [production] | 
            
  | 09:08 | 
  <paravoid> | 
  hard-resetting alsafi, I/O-stuck (qemu bug?) | 
  [production] | 
            
  | 09:05 | 
  <paravoid> | 
  hard-resetting mx1001, I/O-stuck (qemu bug?) | 
  [production] | 
            
  | 08:45 | 
  <moritzm> | 
  upgraded chromium on osmium | 
  [production] | 
            
  | 03:29 | 
  <bblack> | 
  cr1/2-eqiad: re-activating static routes for ns0/ns1 (ipv4/ipv6) pointing to radon -  | 
  [production] | 
            
  | 02:45 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki zhwikisource | 
  [production] | 
            
  | 02:45 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki zhwikinews | 
  [production] | 
            
  | 02:36 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Tue Mar 29 02:36:39 UTC 2016 (duration 8m 46s) | 
  [production] | 
            
  | 02:27 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.18) (duration: 11m 34s) | 
  [production] | 
            
  | 01:07 | 
  <Pchelolo> | 
  restbase finish update to 3ea08751a8 | 
  [production] | 
            
  | 01:00 | 
  <Pchelolo> | 
  restbase start update to 3ea08751a8 | 
  [production] | 
            
  | 00:15 | 
  <krinkle@tin> | 
  Synchronized docroot/wikipedia.org/speed-tests/: (no message) (duration: 00m 32s) | 
  [production] | 
            
  | 00:12 | 
  <mutante> | 
  mw2090 - reinstalled, re-signed puppet/salt | 
  [production] | 
            
  
    | 
      
        2016-03-28
      
      §
     | 
  
    
  | 23:38 | 
  <maxsem@tin> | 
  Synchronized php-1.27.0-wmf.18/includes/api/ApiUpload.php: https://gerrit.wikimedia.org/r/#/c/280092/ (duration: 02m 26s) | 
  [production] | 
            
  | 23:34 | 
  <mutante> | 
  mw2090 - reboot, reinstall as regular appserver | 
  [production] | 
            
  | 23:27 | 
  <maxsem@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280115/ (duration: 00m 35s) | 
  [production] | 
            
  | 23:18 | 
  <maxsem@tin> | 
  Synchronized php-1.27.0-wmf.18/extensions/Echo/: https://gerrit.wikimedia.org/r/#/c/280112/ (duration: 00m 43s) | 
  [production] | 
            
  | 23:16 | 
  <maxsem@tin> | 
  Synchronized php-1.27.0-wmf.18/extensions/MobileApp/: https://gerrit.wikimedia.org/r/#q,279980,n,z (duration: 01m 07s) | 
  [production] | 
            
  | 22:03 | 
  <mutante> | 
  mw2090 - disable puppet/cron, crons now on wasat | 
  [production] | 
            
  | 21:46 | 
  <mutante> | 
  mw1119 restarted hhvm | 
  [production] | 
            
  | 21:26 | 
  <bblack> | 
  http/2 enabled on pinkunicorn.wikimedia.org for testing - T96848 | 
  [production] | 
            
  | 21:26 | 
  <bblack> | 
  http/2 enabled on pinkunicorn.wikimedia.org for testing | 
  [production] | 
            
  | 21:12 | 
  <Pchelolo> | 
  restbase finised deploy of 78e6eab37c | 
  [production] | 
            
  | 21:04 | 
  <Pchelolo> | 
  restbase start deploy of 78e6eab37c | 
  [production] | 
            
  | 21:04 | 
  <mdholloway> | 
  mobileapps deployed 90cfdcd | 
  [production] | 
            
  | 20:37 | 
  <Pchelolo> | 
  restbase canary deploy of 78e6eab37c to restbase1005 | 
  [production] | 
            
  | 20:35 | 
  <mdholloway> | 
  starting mobileapps deployment | 
  [production] | 
            
  | 18:30 | 
  <andrewbogott> | 
  labs public DNS will remain in a transitional state until MarkMonitor updates their records and the TTL expires.  DNS resolution should proceed as usual in the meantime. | 
  [production] | 
            
  | 18:03 | 
  <mutante> | 
  mw1140 restart hhvm | 
  [production] | 
            
  | 17:52 | 
  <cmjohnson1> | 
  shutting down radon to re-apply thermal paste:  | 
  [production] | 
            
  | 17:45 | 
  <mutante> | 
  mw1152,mw2090,terbium  apt-get remove python-mysqldb (T84075) | 
  [production] | 
            
  | 17:44 | 
  <legoktm> | 
  manually fixed T129881 rename | 
  [production] | 
            
  | 17:20 | 
  <paravoid> | 
  cr1/2-eqiad: deactivating static routes for ns0/ns1 (ipv4/ipv6) pointing to radon | 
  [production] | 
            
  | 17:18 | 
  <bblack> | 
  rebooting radon again | 
  [production] | 
            
  | 15:31 | 
  <thcipriani@tin> | 
  Synchronized php-1.27.0-wmf.18/extensions/OpenStackManager/OpenStackManager.php: SWAT: Wikitech: Remove address, domain, proxy special pages [[gerrit:279569]] (duration: 00m 31s) | 
  [production] | 
            
  | 13:32 | 
  <bblack> | 
  rebooting radon (unresponsive console) | 
  [production] | 
            
  | 08:44 | 
  <paravoid> | 
  force-rebooting ms-be2008 to fix hundreds of unkillable mkfs.xfs stuck due to a hosed disk | 
  [production] |