| 2013-02-18
      
      § | 
    
  | 12:33 | <mutante> | wikiartpedia.(biz|co|info|me|mobi|net) activated - redirect to wikipedia.org (RT-4240) | [production] | 
            
  | 12:30 | <mutante> | adding wikiartpedia domains to DNS / authdns-update | [production] | 
            
  | 12:28 | <dzahn> | gracefulled all apaches | [production] | 
            
  | 12:24 | <mutante> | gracefulling eqiad Apaches via dsh for redirect.conf change | [production] | 
            
  | 12:22 | <mutante> | mw1045 - eth0 NO-CARRIER - check cable (RT-4545) | [production] | 
            
  | 12:13 | <mutante> | mw1041 - down - CPU Machine Chk error on mgmt - removing from dsh groups (RT-4517) | [production] | 
            
  | 05:45 | <tstarling> | synchronized wmf-config/db-eqiad.php | [production] | 
            
  | 05:16 | <tstarling> | synchronized wmf-config/db-eqiad.php | [production] | 
            
  | 05:10 | <Tim> | on db1043: running alter table to convert the relevant field to signed, since I figure this is less disruptive than sql_mode = 'NO_UNSIGNED_SUBTRACTION' and less likely to break again than skipping the query | [production] | 
            
  | 04:58 | <Tim> | replication broken on db1043, possibly others, due to update of article_feedback_revisions with negative value for afr_total | [production] | 
            
  | 03:40 | <tstarling> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 02:29 | <LocalisationUpdate> | completed (1.21wmf9) at Mon Feb 18 02:29:47 UTC 2013 | [production] | 
            
  
    | 2013-02-15
      
      § | 
    
  | 23:56 | <RobH> | authdns-update to correct mw110 reverse entry | [production] | 
            
  | 23:28 | <Ryan_Lane> | *modified them in LDAP | [production] | 
            
  | 23:28 | <Ryan_Lane> | modified project objects to indicate whether or not they are using gluster volumes for project and/or home storage | [production] | 
            
  | 23:04 | <RobH> | disregard mw1161 flaps, im working on it | [production] | 
            
  | 20:15 | <Ryan_Lane> | restarting opendj and pdns on virt0 and virt1000 | [production] | 
            
  | 20:15 | <Ryan_Lane> | removed opendj from nfs1/2 | [production] | 
            
  | 19:07 | <RobH> | no one sign mw86-mw125 puppet certs, im still working on them | [production] | 
            
  | 18:23 | <RobH> | mw87-mw125 installing | [production] | 
            
  | 17:33 | <binasher> | reimaging db1029-1031 | [production] | 
            
  | 16:55 | <mutante> | removing mw1085 from dsh groups - broken hardware (RT-4453) | [production] | 
            
  | 14:31 | <mutante> | wikimediafoundation.info activated to redirect .org | [production] | 
            
  | 14:28 | <mutante> | gracefulling Apaches in eqiad using dsh | [production] | 
            
  | 14:25 | <dzahn> | gracefulled all apaches | [production] | 
            
  | 14:15 | <^demon> | [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream | [production] | 
            
  | 08:02 | <olivneh> | synchronized php-1.21wmf9/extensions/Echo  'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4' | [production] | 
            
  | 02:30 | <LocalisationUpdate> | completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013 | [production] | 
            
  | 01:19 | <paravoid> | deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale) | [production] | 
            
  | 01:09 | <olivneh> | synchronized wmf-config/CommonSettings.php  'Enabling Schema NS on test2wiki' | [production] | 
            
  
    | 2013-02-14
      
      § | 
    
  | 22:49 | <RobH> | mw1194 online and now in service as api apache | [production] | 
            
  | 22:48 | <RobH> | mw1200 added to api pool, online and in service | [production] | 
            
  | 22:44 | <RobH> | mw1189-1199 (sans 1194) now online as eqiad api apaches | [production] | 
            
  | 22:37 | <mflaschen> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 22:29 | <mflaschen> | Started syncing Wikimedia installation... : | [production] | 
            
  | 22:21 | <sbernardin> | taking storage1 offline for decommissioning per: https://rt.wikimedia.org/Ticket/Display.html?id=4529 | [production] | 
            
  | 22:18 | <RobH> | pushing mw1161-1188 into apache service (sans 1161/1164) from false to true in pybal, otherwise all online | [production] | 
            
  | 22:16 | <olivneh> | synchronized wmf-config/CommonSettings.php  'Updating  for test2wiki' | [production] | 
            
  | 22:14 | <olivneh> | synchronized php-1.21wmf9/extensions/Math | [production] | 
            
  | 22:03 | <Ryan_Lane> | replacing gluster upstarts on all labstore nodes with a regular init script | [production] | 
            
  | 22:01 | <reedy> | synchronized docroot | [production] | 
            
  | 22:00 | <olivneh> | synchronized wmf-config | [production] | 
            
  | 21:49 | <olivneh> | synchronized wmf-config/InitialiseSettings.php  'Enabling PostEdit on {be|mediawiki|outreach}wikis' | [production] | 
            
  | 21:45 | <olivneh> | synchronized php-1.21wmf9/extensions/EventLogging | [production] | 
            
  | 21:39 | <sbernardin> | taking down srv266 and srv268 for decommissioning ... https://rt.wikimedia.org/Ticket/Display.html?id=4534 | [production] |