901-950 of 10000 results (40ms)
2017-06-29 ยง
14:47 <jynus> several restarts of db2072 services and host on the following hour [production]
14:30 <ema> varnish 4.1.7-1wm1 uploaded to apt.w.o, cp1008 upgraded T164768 [production]
14:08 <marostegui> Deploy alter table on s7 on dbstore1001 - T166208 [production]
13:54 <godog> kick sdb out of mdadm arrays on bast3002 - T169035 [production]
12:56 <akosiaris@tin> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 00m 46s) [production]
12:47 <akosiaris> reboot argon.eqiad.wmnet, darmstadtium.eqiad.wmnet, dbmonitor1001.wikimedia.org, etcd1001.eqiad.wmnet, etcd1006.eqiad.wmnet, krypton.eqiad.wmnet, mendelevium.eqiad.wmnet, mwdebug1001.eqiad.wmnet, roentgenium.eqiad.wmnet, sca1003.eqiad.wmnet for kernel upgrades [production]
12:41 <akosiaris> reboot poolcounter1001 for kernel upgrades [production]
12:38 <marostegui> Stop replication on dbstore1002 - x1 - T169050 [production]
12:29 <akosiaris> reboot nitrogen for kernel upgrades [production]
12:23 <gehel> forcing reindex of cirrus / elasticsearch after switch upgrade [production]
12:23 <akosiaris@tin> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 03m 05s) [production]
12:20 <akosiaris> depool poolcounter1001 for kernel upgrades [production]
12:18 <marostegui> Re-enable event scheduler on dbstore1001 - T169050 [production]
11:58 <marostegui> Stop replication on the same position for: dbstore1001 (s6) and db1050 - T169050 [production]
11:51 <godog> create xfs filesystems on fourth partition on ms-be machines - T151648 [production]
11:48 <ema> cp4015: restart varnish-be [production]
11:31 <ema> route ulsfo back to codfw T168462 [production]
11:09 <ema@neodymium> conftool action : set/ttl=300; selector: dnsdisc=(citoid|restbase-async) [production]
11:06 <ema> repool codfw in DNS after T168462 [production]
11:03 <ema@neodymium> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=citoid [production]
11:02 <ema@neodymium> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=restbase-async [production]
11:02 <ema@neodymium> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=restbase-async [production]
10:57 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment (duration: 02m 56s) [production]
10:54 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment [production]
10:53 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment (duration: 00m 11s) [production]
10:53 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment [production]
10:47 <ema@neodymium> conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid [production]
10:46 <ema@neodymium> conftool action : set/pooled=true; selector: name=codfw,dnsdisc=restbase-async [production]
10:45 <ema> switching citoid and restbase-async back to codfw after T168462 [production]
10:34 <ema> re-enable puppet and start pybal on lvs2001-2003 T168462 [production]
10:30 <ema@neodymium> conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org,service=pdns_recursor [production]
10:30 <ema> repooling acamar T168462 [production]
09:29 <godog> silence paging alerts for *.svc.codfw.wmnet for two hours - T168462 [production]
08:34 <marostegui> Shutdown MySQL and reboot db1034 for maintenance [production]
08:29 <XioNoX> asw-a-codfw upgrade started - T168462 [production]
08:25 <ema> failover codfw LVSs to secondaries T168462 [production]
08:19 <elukey> restart pdfrender on scb1004 - xpra issue [production]
08:16 <volans@neodymium> conftool action : set/pooled=false; selector: name=codfw,dnsdisc=restbase-async [production]
08:15 <volans@neodymium> conftool action : set/pooled=false; selector: name=codfw,dnsdisc=citoid [production]
08:14 <volans@neodymium> conftool action : set/ttl=60; selector: dnsdisc=restbase-async [production]
08:14 <volans@neodymium> conftool action : set/ttl=60; selector: dnsdisc=citoid [production]
08:08 <volans@neodymium> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=citoid [production]
08:07 <volans@neodymium> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=restbase-async [production]
08:05 <ema> bounce pybal on codfw secondary LVSs (lvs2004-2006) [production]
07:57 <volans> switching citoid and restbase-async temporarily to eqiad for T168462 [production]
07:47 <XioNoX> Route cache traffic around codfw - T168462 [production]
07:46 <ema@neodymium> conftool action : set/pooled=no; selector: name=acamar.wikimedia.org,service=pdns_recursor [production]
07:44 <XioNoX> codfw depooled from DNS - T168462 [production]
07:36 <elukey> depooled kafka2001.codfw.wmnet for T168462 [production]
07:19 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment (duration: 02m 55s) [production]