| 
      
        2015-10-28
      
      §
     | 
  
    
  | 15:46 | 
  <andrewbogott> | 
  testing the bot after the nfs move | 
  [production] | 
            
  | 15:40 | 
  <legoktm@tin> | 
  Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/249316 (duration: 00m 18s) | 
  [production] | 
            
  | 15:38 | 
  <legoktm@tin> | 
  Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/249317 (duration: 00m 17s) | 
  [production] | 
            
  | 15:33 | 
  <legoktm@tin> | 
  Synchronized php-1.27.0-wmf.4/extensions/Echo/: https://gerrit.wikimedia.org/r/#/c/249351/ and https://gerrit.wikimedia.org/r/#/c/249411/ (duration: 00m 18s) | 
  [production] | 
            
  | 15:31 | 
  <gwicke> | 
  restarting cassandra on restbase1007 with 20G heap | 
  [production] | 
            
  | 15:30 | 
  <legoktm@tin> | 
  Synchronized php-1.27.0-wmf.4/includes/: LinksUpdate: Keep track of the triggering User - https://gerrit.wikimedia.org/r/#/c/249350/ (duration: 00m 22s) | 
  [production] | 
            
  | 15:26 | 
  <legoktm@tin> | 
  Synchronized wmf-config/Wikibase.php: revert (duration: 00m 17s) | 
  [production] | 
            
  | 15:20 | 
  <legoktm@tin> | 
  Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/249412/ (duration: 00m 17s) | 
  [production] | 
            
  | 15:19 | 
  <legoktm@tin> | 
  Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.search.js: https://gerrit.wikimedia.org/r/#/c/249404/ (duration: 00m 18s) | 
  [production] | 
            
  | 15:15 | 
  <legoktm@tin> | 
  Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.search.js: https://gerrit.wikimedia.org/r/#/c/249405/ (duration: 00m 18s) | 
  [production] | 
            
  | 14:22 | 
  <kart_> | 
  T112626 Finished running fix-stats.php for CX (from rwwiki to zuwiki) | 
  [production] | 
            
  | 14:21 | 
  <moritzm> | 
  repooled restbase1006 | 
  [production] | 
            
  | 14:09 | 
  <moritzm> | 
  depooled restbase1006 for kernel update | 
  [production] | 
            
  | 14:05 | 
  <moritzm> | 
  repooled restbase1005 | 
  [production] | 
            
  | 13:54 | 
  <moritzm> | 
  depooled restbase1005 for kernel update | 
  [production] | 
            
  | 13:47 | 
  <moritzm> | 
  repooled restbase1004 | 
  [production] | 
            
  | 13:35 | 
  <moritzm> | 
  depooled restbase1004 for kernel update | 
  [production] | 
            
  | 13:34 | 
  <moritzm> | 
  repooled restbase1003 | 
  [production] | 
            
  | 13:16 | 
  <moritzm> | 
  depooled restbase1003 for kernel update | 
  [production] | 
            
  | 13:12 | 
  <moritzm> | 
  repooled restbase1002 | 
  [production] | 
            
  | 12:56 | 
  <moritzm> | 
  depooled restbase1002 for kernel update | 
  [production] | 
            
  | 11:46 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 00m 19s) | 
  [production] | 
            
  | 10:39 | 
  <moritzm> | 
  updated kernel on restbase1001 to latest 3.19 | 
  [production] | 
            
  | 10:37 | 
  <_joe_> | 
  manually removed crontab from mw1152, erroneously created by puppet | 
  [production] | 
            
  | 10:11 | 
  <_joe_> | 
  reimaging mw1152 | 
  [production] | 
            
  | 09:15 | 
  <_joe_> | 
  preparing to reimage mw1152, disabling puppet, scheduling downtime. | 
  [production] | 
            
  | 06:22 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Wed Oct 28 06:22:06 UTC 2015 (duration 22m 5s) | 
  [production] | 
            
  | 05:15 | 
  <legoktm> | 
  ran mwscript updateSpecialPages.php --wiki=testwiki --only=GadgetUsage on terbium | 
  [production] | 
            
  | 03:03 | 
  <l10nupdate@tin> | 
  LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-28 03:03:34+00:00 | 
  [production] | 
            
  | 03:00 | 
  <l10nupdate@tin> | 
  Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 00s) | 
  [production] | 
            
  | 02:39 | 
  <l10nupdate@tin> | 
  LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-28 02:39:23+00:00 | 
  [production] | 
            
  | 02:34 | 
  <l10nupdate@tin> | 
  Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 33s) | 
  [production] | 
            
  | 00:55 | 
  <yurik> | 
  deployed graphoid - https://gerrit.wikimedia.org/r/#/c/249324/ | 
  [production] | 
            
  | 00:37 | 
  <tgr@tin> | 
  Finished scap: Updating MediaViewer with r246112 (duration: 43m 18s) | 
  [production] | 
            
  | 00:35 | 
  <mutante> | 
  mw1135 temp. unresponsive - OOM killer killing hhvm  | 
  [production] | 
            
  | 00:27 | 
  <mutante> | 
  powercycling unresponsive mw1127 | 
  [production] | 
            
  
    | 
      
        2015-10-27
      
      §
     | 
  
    
  | 23:53 | 
  <tgr@tin> | 
  Started scap: Updating MediaViewer with r246112 | 
  [production] | 
            
  | 23:13 | 
  <ori> | 
  running sync-common on mw2050, mw2128, mw2187 and mw2209 (cf I324134438955c7) | 
  [production] | 
            
  | 23:12 | 
  <gwicke> | 
  ran `sudo mdadm --readwrite md1` on restbase1007 to resolve `pending` state | 
  [production] | 
            
  | 23:11 | 
  <gwicke> | 
  rebooted restbase1007 to rule out a funky hardware state causing elevated read latencies | 
  [production] | 
            
  | 23:05 | 
  <krenair@tin> | 
  Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s) | 
  [production] | 
            
  | 23:05 | 
  <krenair@tin> | 
  Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s) | 
  [production] | 
            
  | 23:05 | 
  <krenair@tin> | 
  Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 18s) | 
  [production] | 
            
  | 23:01 | 
  <legoktm@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/249306, no-op (duration: 00m 18s) | 
  [production] | 
            
  | 22:39 | 
  <mutante> | 
  powercycling unresponsive analytics1039, here's what i saw on mgmt https://phabricator.wikimedia.org/P2248 | 
  [production] | 
            
  | 21:50 | 
  <aaron@tin> | 
  Synchronized php-1.27.0-wmf.4/includes/MovePage.php: 7a7c7b27d6c (duration: 00m 17s) | 
  [production] | 
            
  | 21:50 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.3/extensions/Flow/: Fix for cache key bug (duration: 00m 20s) | 
  [production] | 
            
  | 21:29 | 
  <twentyafterfour@tin> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.4 | 
  [production] | 
            
  | 21:29 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.3/includes/debug/logger/LoggerFactory.php: I437bcb532: LoggerFactory: Only check for Psr\Log\LoggerInterface once (duration: 00m 18s) | 
  [production] | 
            
  | 19:46 | 
  <twentyafterfour@tin> | 
  Finished scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch (duration: 31m 32s) | 
  [production] |