1451-1500 of 10000 results (34ms)
2021-02-15 §
07:20 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-tool1008.eqiad.wmnet [production]
07:16 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet [production]
07:14 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet [production]
07:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1162 with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14341 and previous config saved to /var/cache/conftool/dbconfig/20210215-070206-marostegui.json [production]
06:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1162 with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14340 and previous config saved to /var/cache/conftool/dbconfig/20210215-064628-marostegui.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1162 to dbctl - depooled T258361', diff saved to https://phabricator.wikimedia.org/P14339 and previous config saved to /var/cache/conftool/dbconfig/20210215-064001-marostegui.json [production]
2021-02-14 §
13:13 <akosiaris> sudo cumin -b 1 -s 120 'cp500[2,3,5,6].eqsin.wmnet' 'systemctl restart varnish-frontend.service' [production]
13:10 <_joe_> restarted varnish-fe on cp5004 [production]
13:09 <akosiaris> restart varnish-fe on cp5001 [production]
09:27 <joal@deploy1001> Finished deploy [analytics/refinery@dd5f947] (thin): Hotfix analytics deployment - THIN [analytics/refinery@dd5f947] (duration: 00m 06s) [production]
09:27 <joal@deploy1001> Started deploy [analytics/refinery@dd5f947] (thin): Hotfix analytics deployment - THIN [analytics/refinery@dd5f947] [production]
09:27 <joal@deploy1001> Finished deploy [analytics/refinery@dd5f947]: Hotfix analytics deployment [analytics/refinery@dd5f947] (duration: 12m 52s) [production]
09:14 <joal@deploy1001> Started deploy [analytics/refinery@dd5f947]: Hotfix analytics deployment [analytics/refinery@dd5f947] [production]
2021-02-13 §
03:23 <ryankemper> Depooled `wdqs1006` to catch up on lag [production]
03:23 <ryankemper> Restarted blazegraph on `wdqs1006` [production]
01:30 <crusnov@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host mwdebug1002.eqiad.wmnet [production]
01:00 <crusnov@cumin1001> START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet [production]
00:55 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet [production]
00:49 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet [production]
00:38 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet [production]
00:31 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1281.eqiad.wmnet [production]
00:31 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1282.eqiad.wmnet [production]
00:31 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1283.eqiad.wmnet [production]
00:31 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1284.eqiad.wmnet [production]
00:30 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet [production]
00:30 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet [production]
00:26 <mutante> ganeti - attempting to recreate VM mwdebug1002 with cookbook that wsa previously deleted manually (T274689 T274023) [production]
00:25 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet [production]
00:08 <mutante> ganeti1011 - manually deleting VM mwdebug1002 - T274689 T274023 [production]
2021-02-12 §
23:59 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1348.eqiad.wmnet [production]
23:58 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1356.eqiad.wmnet [production]
23:43 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1284.eqiad.wmnet [production]
23:42 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1283.eqiad.wmnet [production]
23:42 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1282.eqiad.wmnet [production]
23:42 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1348.eqiad.wmnet [production]
23:41 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw1356.eqiad.wmnet [production]
23:41 <legoktm@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1221.eqiad.wmnet [production]
23:39 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1221.eqiad.wmnet [production]
23:38 <legoktm@cumin1001> conftool action : set/pooled=no; selector: name=mw1281.eqiad.wmnet [production]
23:26 <dduvall@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:24 <dduvall@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:14 <dduvall@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
23:02 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
22:52 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
22:51 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1284.eqiad.wmnet with reason: REIMAGE [production]
22:49 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1283.eqiad.wmnet with reason: REIMAGE [production]
22:48 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1284.eqiad.wmnet with reason: REIMAGE [production]
22:47 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1283.eqiad.wmnet with reason: REIMAGE [production]
22:47 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1282.eqiad.wmnet with reason: REIMAGE [production]
22:45 <legoktm@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1282.eqiad.wmnet with reason: REIMAGE [production]