2201-2250 of 10000 results (51ms)
2022-06-28 §
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1149 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30509 and previous config saved to /var/cache/conftool/dbconfig/20220628-071001-root.json [production]
07:05 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
07:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
07:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T309311)', diff saved to https://phabricator.wikimedia.org/P30508 and previous config saved to /var/cache/conftool/dbconfig/20220628-070525-ladsgroup.json [production]
07:02 <marostegui> dbmaint s7@eqiad T302659 [production]
07:02 <marostegui> dbmaint s7@eqiad T310011 [production]
06:00 <marostegui> Starting s7 eqiad failover from db1136 to db1181 - T311033 [production]
05:29 <marostegui> dbmaint s6@codfw T298557 [production]
2022-06-27 §
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1474.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1470.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1468.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1476.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1472.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1471.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1469.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1475.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1467.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:51 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1473.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1457.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1461.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1464.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1460.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1465.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1458.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1459.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1462.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1466.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw1463.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1465.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1464.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1457.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1462.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1463.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1459.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1460.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1466.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host mw1458.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:22 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:18 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
22:36 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:30 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
21:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T309311)', diff saved to https://phabricator.wikimedia.org/P30491 and previous config saved to /var/cache/conftool/dbconfig/20220627-215105-ladsgroup.json [production]
21:36 <urbanecm> wikiadmin@10.64.16.184(cswiki)> delete from user_properties where up_property='growthexperiments-homepage-suggestededits-topics-enabled'; # T308309 [production]
21:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P30490 and previous config saved to /var/cache/conftool/dbconfig/20220627-213600-ladsgroup.json [production]
21:35 <urbanecm> wikiadmin@10.64.48.109(viwiki)> delete from user_properties where up_property='growthexperiments-homepage-suggestededits-topics-enabled'; # T308309 [production]
21:35 <urbanecm> wikiadmin@10.64.48.109(arwiki)> delete from user_properties where up_property='growthexperiments-homepage-suggestededits-topics-enabled'; # T308309 [production]
21:34 <urbanecm> wikiadmin@10.64.48.109(kowiki)> delete from user_properties where up_property='growthexperiments-homepage-suggestededits-topics-enabled'; # T308309 [production]
21:23 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:22 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]