1301-1350 of 10000 results (32ms)
2021-02-09 §
06:47 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@582b070]: 0.3.63 (duration: 06m 46s) [production]
06:44 <XioNoX> depool esams for network maintenance - T272342 [production]
06:41 <ryankemper> [WDQS Deploy] Tests passing following deploy of `0.3.63` on canary `wdqs1003`; proceeding to rest of fleet [production]
06:40 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@582b070]: 0.3.63 [production]
06:40 <ryankemper> Pooled `wdqs1007` and depooled `wdqs1005` (`1005` is ~12 hours behind) [production]
06:38 <ryankemper> [WDQS Deploy] Gearing up for deploy of wdqs `0.3.63`. Pre-deploy tests passing on canary `wdqs1003` [production]
06:35 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 25%: Slowly repooling db1111 after onsite maintenance', diff saved to https://phabricator.wikimedia.org/P14237 and previous config saved to /var/cache/conftool/dbconfig/20210209-063527-root.json [production]
06:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 10%: Slowly repooling db1111 after onsite maintenance', diff saved to https://phabricator.wikimedia.org/P14236 and previous config saved to /var/cache/conftool/dbconfig/20210209-062024-root.json [production]
06:20 <marostegui> Stop mysql on s2 and s7 on db1090 to clone db1170 T258361 [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1090:3312, db1090:3317 T258361', diff saved to https://phabricator.wikimedia.org/P14234 and previous config saved to /var/cache/conftool/dbconfig/20210209-061822-marostegui.json [production]
06:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db1111 (re)pooling @ 5%: Slowly repooling db1111 after onsite maintenance', diff saved to https://phabricator.wikimedia.org/P14233 and previous config saved to /var/cache/conftool/dbconfig/20210209-060520-root.json [production]
05:02 <krinkle@deploy1001> Finished deploy [integration/docroot@fdfb265]: I271e6054880, T273247 (duration: 00m 06s) [production]
05:02 <krinkle@deploy1001> Started deploy [integration/docroot@fdfb265]: I271e6054880, T273247 [production]
01:56 <tstarling@deploy1001> Synchronized php-1.36.0-wmf.29/extensions/FeaturedFeeds: probable fix for UBN T273242 (duration: 01m 06s) [production]
01:46 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1302.eqiad.wmnet [production]
01:46 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1301.eqiad.wmnet [production]
00:48 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1302.eqiad.wmnet [production]
00:48 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1301.eqiad.wmnet [production]
00:28 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1387.eqiad.wmnet [production]
00:24 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1386.eqiad.wmnet [production]
00:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1386.eqiad.wmnet [production]
00:22 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1387.eqiad.wmnet [production]
00:02 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1301.eqiad.wmnet with reason: REIMAGE [production]
00:00 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1302.eqiad.wmnet with reason: REIMAGE [production]
2021-02-08 §
23:59 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1301.eqiad.wmnet with reason: REIMAGE [production]
23:58 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1302.eqiad.wmnet with reason: REIMAGE [production]
23:52 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on mw2220.codfw.wmnet with reason: T273803 [production]
23:52 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on mw2220.codfw.wmnet with reason: T273803 [production]
23:50 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw2220.codfw.wmnet [production]
23:49 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2220.codfw.wmnet [production]
23:49 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1386.eqiad.wmnet with reason: REIMAGE [production]
23:47 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1386.eqiad.wmnet with reason: REIMAGE [production]
23:46 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1387.eqiad.wmnet with reason: REIMAGE [production]
23:44 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1387.eqiad.wmnet with reason: REIMAGE [production]
23:29 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1388.eqiad.wmnet [production]
23:17 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1303.eqiad.wmnet [production]
23:16 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2245.codfw.wmnet [production]
23:14 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1397.eqiad.wmnet [production]
22:51 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1274.eqiad.wmnet [production]
22:51 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1273.eqiad.wmnet [production]
22:51 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1272.eqiad.wmnet [production]
22:51 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1271.eqiad.wmnet [production]
22:46 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2245.codfw.wmnet [production]
22:46 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1388.eqiad.wmnet [production]
22:45 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1303.eqiad.wmnet [production]
22:45 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1397.eqiad.wmnet [production]
22:29 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1274.eqiad.wmnet [production]
22:29 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1273.eqiad.wmnet [production]
22:29 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1272.eqiad.wmnet [production]
22:29 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1271.eqiad.wmnet [production]