|
2023-04-11
ยง
|
| 07:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 25%: Pooling', diff saved to https://phabricator.wikimedia.org/P46273 and previous config saved to /var/cache/conftool/dbconfig/20230411-074006-root.json |
[production] |
| 07:39 |
<jelto@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab2003.wikimedia.org with OS bullseye |
[production] |
| 07:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1103.eqiad.wmnet |
[production] |
| 07:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1103.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 07:37 |
<marostegui@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1103.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 3%: Pooling', diff saved to https://phabricator.wikimedia.org/P46272 and previous config saved to /var/cache/conftool/dbconfig/20230411-073651-root.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 10%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46271 and previous config saved to /var/cache/conftool/dbconfig/20230411-073319-root.json |
[production] |
| 07:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1103.eqiad.wmnet |
[production] |
| 07:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P46270 and previous config saved to /var/cache/conftool/dbconfig/20230411-073047-root.json |
[production] |
| 07:30 |
<dcausse> |
restarting blazegraph on wdqs1007 (stuck for 48hours) |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 10%: Pooling', diff saved to https://phabricator.wikimedia.org/P46269 and previous config saved to /var/cache/conftool/dbconfig/20230411-072501-root.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 2%: Pooling', diff saved to https://phabricator.wikimedia.org/P46268 and previous config saved to /var/cache/conftool/dbconfig/20230411-072146-root.json |
[production] |
| 07:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 5%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46267 and previous config saved to /var/cache/conftool/dbconfig/20230411-071815-root.json |
[production] |
| 07:18 |
<zabe@deploy2002> |
Finished scap: Backport for [[gerrit:906793|Add blkwiki to wgSitename (T334351)]] (duration: 08m 08s) |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1103 from dbctl T332293', diff saved to https://phabricator.wikimedia.org/P46266 and previous config saved to /var/cache/conftool/dbconfig/20230411-071647-marostegui.json |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P46265 and previous config saved to /var/cache/conftool/dbconfig/20230411-071542-root.json |
[production] |
| 07:11 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 393731 |
[production] |
| 07:11 |
<zabe@deploy2002> |
zabe and jhsoby: Backport for [[gerrit:906793|Add blkwiki to wgSitename (T334351)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 07:11 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 393731 |
[production] |
| 07:11 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 150279 |
[production] |
| 07:10 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 150279 |
[production] |
| 07:10 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 35467 |
[production] |
| 07:10 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 35467 |
[production] |
| 07:10 |
<zabe@deploy2002> |
Started scap: Backport for [[gerrit:906793|Add blkwiki to wgSitename (T334351)]] |
[production] |
| 07:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 5%: Pooling', diff saved to https://phabricator.wikimedia.org/P46264 and previous config saved to /var/cache/conftool/dbconfig/20230411-070956-root.json |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 1%: Pooling', diff saved to https://phabricator.wikimedia.org/P46263 and previous config saved to /var/cache/conftool/dbconfig/20230411-070641-root.json |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1211 to dbctl T326669', diff saved to https://phabricator.wikimedia.org/P46262 and previous config saved to /var/cache/conftool/dbconfig/20230411-070609-marostegui.json |
[production] |
| 07:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 4%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46261 and previous config saved to /var/cache/conftool/dbconfig/20230411-070310-root.json |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P46260 and previous config saved to /var/cache/conftool/dbconfig/20230411-070037-root.json |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1118 T334375', diff saved to https://phabricator.wikimedia.org/P46258 and previous config saved to /var/cache/conftool/dbconfig/20230411-065734-marostegui.json |
[production] |
| 06:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1163 to s1 primary T334375', diff saved to https://phabricator.wikimedia.org/P46257 and previous config saved to /var/cache/conftool/dbconfig/20230411-065639-root.json |
[production] |
| 06:56 |
<marostegui> |
Starting s1 eqiad failover from db1118 to db1163 - T334375 |
[production] |
| 06:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 4%: Pooling', diff saved to https://phabricator.wikimedia.org/P46256 and previous config saved to /var/cache/conftool/dbconfig/20230411-065452-root.json |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 3%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46255 and previous config saved to /var/cache/conftool/dbconfig/20230411-064805-root.json |
[production] |
| 06:43 |
<jelto@cumin2002> |
START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye |
[production] |
| 06:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 3%: Pooling', diff saved to https://phabricator.wikimedia.org/P46254 and previous config saved to /var/cache/conftool/dbconfig/20230411-063947-root.json |
[production] |
| 06:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 2%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46252 and previous config saved to /var/cache/conftool/dbconfig/20230411-063300-root.json |
[production] |
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 2%: Pooling', diff saved to https://phabricator.wikimedia.org/P46251 and previous config saved to /var/cache/conftool/dbconfig/20230411-062442-root.json |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1163 with weight 0 T334375', diff saved to https://phabricator.wikimedia.org/P46250 and previous config saved to /var/cache/conftool/dbconfig/20230411-062127-root.json |
[production] |
| 06:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 37 hosts with reason: Primary switchover s1 T334375 |
[production] |
| 06:20 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 37 hosts with reason: Primary switchover s1 T334375 |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 1%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46249 and previous config saved to /var/cache/conftool/dbconfig/20230411-061755-root.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1209 to dbctl T326206', diff saved to https://phabricator.wikimedia.org/P46248 and previous config saved to /var/cache/conftool/dbconfig/20230411-061642-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1110 to clone db1210 T326669', diff saved to https://phabricator.wikimedia.org/P46246 and previous config saved to /var/cache/conftool/dbconfig/20230411-061044-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 1%: Pooling', diff saved to https://phabricator.wikimedia.org/P46245 and previous config saved to /var/cache/conftool/dbconfig/20230411-060937-root.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1224 to dbctl T326206', diff saved to https://phabricator.wikimedia.org/P46244 and previous config saved to /var/cache/conftool/dbconfig/20230411-060922-marostegui.json |
[production] |
| 05:45 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Swakiyama out of all services on: 814 hosts |
[production] |
| 05:45 |
<jmm@cumin2002> |
START - Cookbook sre.idm.logout Logging Swakiyama out of all services on: 814 hosts |
[production] |