| 
      
        2016-07-05
      
      ยง
     | 
  
    
  | 20:00 | 
  <akosiaris> | 
  T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-apy_0.9.1~r343-1 | 
  [production] | 
            
  | 18:54 | 
  <chasemp> | 
  add dpatrick to wmf-nda | 
  [production] | 
            
  | 18:32 | 
  <Danny_B> | 
  killed wb2-phab @ tools.wikibugs to perform some batch edits | 
  [production] | 
            
  | 18:30 | 
  <mutante> | 
  added siddharth11 to LDAP group "wmf" per T138369#2425011 | 
  [production] | 
            
  | 16:44 | 
  <jynus> | 
  rebooting pc2006 T139283 | 
  [production] | 
            
  | 16:20 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes; selector: mw1024.eqiad.wmnet | 
  [production] | 
            
  | 16:13 | 
  <elukey@palladium> | 
  conftool action : set/pooled=no; selector: mw1024.eqiad.wmnet | 
  [production] | 
            
  | 16:12 | 
  <elukey> | 
  depooling mw1024 to restore regular fcgi logging settings | 
  [production] | 
            
  | 16:09 | 
  <urandom> | 
  Bootstrapping restbase1009-c.eqiad.wmnet : T139362 | 
  [production] | 
            
  | 15:49 | 
  <akosiaris> | 
  reenable alerts from smokeping on codfw | 
  [production] | 
            
  | 15:46 | 
  <andrewbogott> | 
  rebooting labservices1002 to see if it survives the reboot better this time | 
  [production] | 
            
  | 15:32 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: touch initialisesettings to clear logspam, hopefully (duration: 00m 27s) | 
  [production] | 
            
  | 15:20 | 
  <elukey> | 
  powercycled mw1140, memory saturated and not reachable via ssh/mgmt-console | 
  [production] | 
            
  | 15:20 | 
  <paravoid> | 
  mr1-codfw: "request system snapshot media internal slice alternate" + "request system reboot" | 
  [production] | 
            
  | 15:08 | 
  <yurik> | 
  depl & restarted tilerator https://gerrit.wikimedia.org/r/#/c/297410/ | 
  [production] | 
            
  | 14:42 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes:weight=25; selector: mw1283.eqiad.wmnet | 
  [production] | 
            
  | 14:41 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes:weight=25; selector: mw1282.eqiad.wmnet | 
  [production] | 
            
  | 14:37 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes:weight=25; selector: mw1281.eqiad.wmnet | 
  [production] | 
            
  | 14:26 | 
  <urandom> | 
  Issuing `nodetool cleanup' for restbase1014-b.eqiad.wmnet | 
  [production] | 
            
  | 14:12 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes:weight=25; selector: mw1280.eqiad.wmnet | 
  [production] | 
            
  | 14:09 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes:weight=25; selector: mw1279.eqiad.wmnet | 
  [production] | 
            
  | 14:07 | 
  <oblivian@palladium> | 
  conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=appserver,service=apache2,name=mw1217.* | 
  [production] | 
            
  | 14:07 | 
  <elukey> | 
  Pooling first batch of new eqiad api-servers - mw1279->mw1283 | 
  [production] | 
            
  | 14:03 | 
  <_joe_> | 
  depooling permanently mw1091-13 from the appservers pool in eqiad | 
  [production] | 
            
  | 13:58 | 
  <elukey@palladium> | 
  conftool action : set/weight=20; selector: mw2245.codfw.wmnet | 
  [production] | 
            
  | 13:58 | 
  <elukey@palladium> | 
  conftool action : set/weight=20; selector: mw2244.codfw.wmnet | 
  [production] | 
            
  | 13:58 | 
  <elukey@palladium> | 
  conftool action : set/weight=20; selector: mw2243.codfw.wmnet | 
  [production] | 
            
  | 13:58 | 
  <elukey@palladium> | 
  conftool action : set/weight=20; selector: mw2242.codfw.wmnet | 
  [production] | 
            
  | 13:57 | 
  <elukey@palladium> | 
  conftool action : set/weight=20; selector: mw2241.codfw.wmnet | 
  [production] | 
            
  | 13:57 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes; selector: mw2245.codfw.wmnet | 
  [production] | 
            
  | 13:57 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes; selector: mw2244.codfw.wmnet | 
  [production] | 
            
  | 13:57 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes; selector: mw2243.codfw.wmnet | 
  [production] | 
            
  | 13:56 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes; selector: mw2242.codfw.wmnet | 
  [production] | 
            
  | 13:56 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes; selector: mw2241.codfw.wmnet | 
  [production] | 
            
  | 13:56 | 
  <elukey> | 
  pooling new codfw appservers - mw224[12345] | 
  [production] | 
            
  | 12:32 | 
  <elukey@palladium> | 
  conftool action : set/pooled=yes; selector: mw1024.eqiad.wmnet | 
  [production] | 
            
  | 12:12 | 
  <elukey@palladium> | 
  conftool action : set/pooled=no; selector: mw1024.eqiad.wmnet | 
  [production] | 
            
  | 12:11 | 
  <elukey> | 
  depooling/re-pooling mw1024.eqiad.wmnet to temporarily set up trace8 logging (503 investigation - T73487) | 
  [production] | 
            
  | 12:08 | 
  <jynus> | 
  running schema change on db1019 T73563 | 
  [production] | 
            
  | 11:15 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Failover all commons special roles to db1081 (duration: 00m 24s) | 
  [production] | 
            
  | 11:00 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Failover commons recentachanges (duration: 00m 36s) | 
  [production] | 
            
  | 10:45 | 
  <jynus> | 
  SET GLOBAL read_only=0; on db1040, our new m4-master | 
  [production] | 
            
  | 10:38 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Failover commons master to db1040 (duration: 00m 32s) | 
  [production] | 
            
  | 10:23 | 
  <jynus> | 
  archiving m3-master phlegal* databases before dropping them | 
  [production] | 
            
  | 10:20 | 
  <mobrovac> | 
  restbase staging started a no-op dump on cerium to test restbase on node 4.4.6 | 
  [production] | 
            
  | 10:05 | 
  <elukey@palladium> | 
  conftool action : set/weight=30; selector: mw1275.eqiad.wmnet | 
  [production] | 
            
  | 10:05 | 
  <elukey@palladium> | 
  conftool action : set/weight=30; selector: mw1274.eqiad.wmnet | 
  [production] | 
            
  | 10:05 | 
  <elukey@palladium> | 
  conftool action : set/weight=30; selector: mw1273.eqiad.wmnet | 
  [production] | 
            
  | 09:59 | 
  <elukey@palladium> | 
  conftool action : set/weight=30; selector: mw1272.eqiad.wmnet | 
  [production] | 
            
  | 09:31 | 
  <_joe_> | 
  shutting down mw1009-16 for decommissioning | 
  [production] |