601-650 of 10000 results (59ms)
2019-05-17 §
07:57 <fsero> disabling puppet on mcrouter hosts for T221346 [production]
07:11 <marostegui> Compress s7 on labsdb1012 T222978 [production]
06:36 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 49s) [production]
06:35 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db2111 and db2113 into s5 T222772 (duration: 00m 50s) [production]
05:19 <marostegui> Stop MySQL on db1083 to clone db1134 [production]
05:17 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 50s) [production]
05:00 <mobrovac> bootstrap 1021-a - T219404 [production]
2019-05-16 §
21:02 <Jeff_Green> authdns-update to switch payments.wikimedia.org back to eqiad cluster [production]
19:24 <onimisionipe> pooling elastic2038 - shards are properly balanced across nodes [production]
18:31 <onimisionipe> depooling elastic2038 to investigate more [production]
17:26 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:26 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:26 <jbond42> reboot ores1007-1009 [production]
17:15 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:15 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:15 <jbond42> reboot ores1005-1006 [production]
17:10 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:10 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:10 <jbond42> reboot ores1003-1004 [production]
17:05 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:05 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:05 <jbond42> reboot ores1001-1002 [production]
17:00 <jbond42> reboot orespoolcounter[12]002 [production]
16:53 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:53 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:53 <jbond@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
16:53 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:53 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:52 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:52 <jbond@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
16:52 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:51 <jbond42> reboot orespoolcounter[12]001 [production]
16:44 <jbond42> reboot ores2008-2009 [production]
16:38 <jbond42> will frist reboot ores2006-2007 [production]
16:36 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:36 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:36 <jbond42> reboot ores2006-2009 [production]
16:28 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:28 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:28 <jbond42> reboot ores2003-2005 [production]
16:22 <XioNoX> add BGP session to Hetzner in AMS-IX [production]
16:19 <akosiaris> switch all etcd* kubestagetcd* servers from "drbd" ganeti disk template to "plain" ganeti disk template [production]
16:17 <jbond42> reboot ores2001-2002 [production]
16:16 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:16 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:59 <akosiaris> build service-checker OCI container 0.0.2 with 0.1.5 service-checker version T220401 [production]
15:49 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.5/extensions/CirrusSearch/includes/InterwikiSearcher.php: Hot-deploy CirrusSearch interwiki no result UBN T223449 (duration: 00m 49s) [production]
15:45 <marostegui> Drop the following databases from tendril to recreated them with the right user: db1127,db1129,db1130, db1131, db1137,db1138 [production]
15:35 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.5/includes/specials/pagers/ContribsPager.php: Hot-deploy Contribs getNamespaceInfo UBN fix T223440 (duration: 00m 53s) [production]
15:25 <aborrero@puppetmaster1001> conftool action : set/pooled=yes; selector: name=labweb1001.wikimedia.org,service=labweb [production]