| 
      
        2015-11-18
      
      §
     | 
  
    
  | 17:30 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.6/includes/db/loadbalancer/LoadBalancer.php: I03e1386b: Make getLaggedSlaveMode() use reuseConnection() as needed (T118162) (duration: 00m 19s) | 
  [production] | 
            
  | 17:26 | 
  <andrewbogott> | 
  repooled mw1154, depooled mw1155 | 
  [production] | 
            
  | 17:17 | 
  <andrewbogott> | 
  repooled mw1153, depooling mw1154 | 
  [production] | 
            
  | 17:09 | 
  <andrewbogott> | 
  depooled mw1153 for https://phabricator.wikimedia.org/T118888 | 
  [production] | 
            
  | 17:07 | 
  <thcipriani> | 
  updateCollation.php for srwiki 1670608 rows processed | 
  [production] | 
            
  | 17:05 | 
  <godog> | 
  restart restbase2002-a cassandra instance, bootstrap failed with stream error | 
  [production] | 
            
  | 16:23 | 
  <bblack> | 
  upgrading pybal on lvs1006 (1.13.1) | 
  [production] | 
            
  | 16:23 | 
  <_joe_> | 
  upgrading pybal on lvs1006 (1.13.1) | 
  [production] | 
            
  | 16:21 | 
  <bblack> | 
  upgrading pybal on lvs1005 (1.13.1) | 
  [production] | 
            
  | 16:21 | 
  <bblack> | 
  upgrading pybal on lvs1004 (1.13.1) | 
  [production] | 
            
  | 16:16 | 
  <thcipriani> | 
  updateCollation.php for srwiktionary complete 82713 rows processed | 
  [production] | 
            
  | 16:12 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation to "uca-sr" at srwiki and srwiktionary [[gerrit:253903]] (duration: 00m 20s) | 
  [production] | 
            
  | 16:11 | 
  <bblack> | 
  upgrading pybal on lvs1004 (1.13.1) | 
  [production] | 
            
  | 16:07 | 
  <mobrovac> | 
  restbase cassandra dropping old mobileapps keyspaces | 
  [production] | 
            
  | 16:05 | 
  <mobrovac> | 
  restbase end deploy of 6b5a602 | 
  [production] | 
            
  | 15:56 | 
  <mobrovac> | 
  restbase start deploy of 6b5a602 | 
  [production] | 
            
  | 15:50 | 
  <bblack> | 
  upgrading pybal to 1.13 on lvs100[56] (with service clean) | 
  [production] | 
            
  | 15:37 | 
  <jynus> | 
  warming up srwiki.page and srwiki.categorylinks on s3 slaves | 
  [production] | 
            
  | 15:29 | 
  <mobrovac> | 
  restbase canary deploy to rb1001 of 6b5a602 | 
  [production] | 
            
  | 15:28 | 
  <bblack> | 
  upgrading pybal to 1.13 on lvs1004, with cleared tables | 
  [production] | 
            
  | 15:23 | 
  <_joe_> | 
  uploaded pybal 1.13 to jessie-wikimedia | 
  [production] | 
            
  | 15:14 | 
  <hashar> | 
  stopping zuul-merger on scandium. Lacks ssh private key to reach gerrit | 
  [production] | 
            
  | 14:58 | 
  <hashar> | 
  Pooled zuul-merger instance on scandium.eqiad.wmnet . In case it screw up one has to stop the zuul-merger service on scandium. | 
  [production] | 
            
  | 14:57 | 
  <godog> | 
  bounce mobileapps on scb1001/scb1002 | 
  [production] | 
            
  | 14:47 | 
  <mobrovac> | 
  mobileapps deploying 151c312 | 
  [production] | 
            
  | 13:55 | 
  <bblack> | 
  disabling pybal on lvs100[123] - fallback to lvs100[456] | 
  [production] | 
            
  | 12:00 | 
  <hashar> | 
  reenabled puppet on gallium. Havent pooled zuul-merger yet | 
  [production] | 
            
  | 11:57 | 
  <hashar> | 
  disabled puppet on gallium | 
  [production] | 
            
  | 11:56 | 
  <hashar> | 
  pooling new zuul-merger on scandium | 
  [production] | 
            
  | 11:44 | 
  <mark> | 
  Set OSPF/OSPFv3 metric 320 (instead of 360) on new Zayo link between cr2-eqiad:xe-5/2/3 and cr2-codfw:xe-5/0/1 | 
  [production] | 
            
  | 10:49 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Decreasing weight of db1035 due to connection exhaustion (duration: 00m 19s) | 
  [production] | 
            
  | 10:31 | 
  <hashar> | 
  gallium: apt-get upgrade | 
  [production] | 
            
  | 10:30 | 
  <hashar> | 
  upgrading Zuul on gallium (CI interruption for a minute) | 
  [production] | 
            
  | 09:48 | 
  <jynus> | 
  several db maintenance tasks on db1044 -optimization, upgrade- expect lag (node is depooled) | 
  [production] | 
            
  | 09:25 | 
  <godog> | 
  reimage restbase2002 | 
  [production] | 
            
  | 09:11 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1027, depool db1044 for maintenance (duration: 01m 08s) | 
  [production] | 
            
  | 09:09 | 
  <_joe_> | 
  restarted HHVM on mw1095, stuck in treadmill::startrequest() on a contended lock | 
  [production] | 
            
  | 06:31 | 
  <_joe_> | 
  rsyncing terbium homes from rutherfordium | 
  [production] | 
            
  | 02:39 | 
  <l10nupdate@tin> | 
  Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 10m 10s) | 
  [production] | 
            
  | 01:46 | 
  <andrewbogott> | 
  repooling mw1156 | 
  [production] | 
            
  | 01:43 | 
  <yurik_> | 
  synced graphoid service | 
  [production] | 
            
  | 01:40 | 
  <andrewbogott> | 
  updated kernel on mw1156, now rebooting. | 
  [production] | 
            
  | 01:35 | 
  <andrewbogott> | 
  repooling mw1158, depooling mw1156 | 
  [production] | 
            
  | 01:22 | 
  <andrewbogott> | 
  rebooting mw1158 | 
  [production] | 
            
  | 01:19 | 
  <andrewbogott> | 
  upgrading mw1158 to 3.13.0-62 | 
  [production] | 
            
  | 00:26 | 
  <legoktm@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: wgRCWatchCategoryMembership true on group0 wikis (duration: 00m 27s) | 
  [production] | 
            
  | 00:04 | 
  <maxsem@tin> | 
  Synchronized portals/: SWAT (duration: 00m 32s) | 
  [production] |