|
2023-04-11
ยง
|
| 09:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 1%: Pooling', diff saved to https://phabricator.wikimedia.org/P46292 and previous config saved to /var/cache/conftool/dbconfig/20230411-090310-root.json |
[production] |
| 08:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1122 to clone db1222 T326669', diff saved to https://phabricator.wikimedia.org/P46290 and previous config saved to /var/cache/conftool/dbconfig/20230411-085654-marostegui.json |
[production] |
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 50%: Pooling', diff saved to https://phabricator.wikimedia.org/P46289 and previous config saved to /var/cache/conftool/dbconfig/20230411-085215-root.json |
[production] |
| 08:50 |
<jelto@cumin2002> |
START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye |
[production] |
| 08:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 25%: Pooling', diff saved to https://phabricator.wikimedia.org/P46288 and previous config saved to /var/cache/conftool/dbconfig/20230411-083710-root.json |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 100%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46287 and previous config saved to /var/cache/conftool/dbconfig/20230411-083339-root.json |
[production] |
| 08:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P46286 and previous config saved to /var/cache/conftool/dbconfig/20230411-083106-root.json |
[production] |
| 08:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 100%: Pooling', diff saved to https://phabricator.wikimedia.org/P46285 and previous config saved to /var/cache/conftool/dbconfig/20230411-082521-root.json |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 10%: Pooling', diff saved to https://phabricator.wikimedia.org/P46284 and previous config saved to /var/cache/conftool/dbconfig/20230411-082205-root.json |
[production] |
| 08:19 |
<aqu@deploy2002> |
Finished deploy [analytics/refinery@bed78f6] (hadoop-test): Deploy analytics_refinery including last webrequest load scripts in TEST 3nd try [analytics/refinery@bed78f6] (duration: 01m 25s) |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 75%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46283 and previous config saved to /var/cache/conftool/dbconfig/20230411-081834-root.json |
[production] |
| 08:18 |
<aqu@deploy2002> |
Started deploy [analytics/refinery@bed78f6] (hadoop-test): Deploy analytics_refinery including last webrequest load scripts in TEST 3nd try [analytics/refinery@bed78f6] |
[production] |
| 08:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P46282 and previous config saved to /var/cache/conftool/dbconfig/20230411-081601-root.json |
[production] |
| 08:15 |
<aqu> |
About to deploy analytics/refinery (To migrate webrequest load from Oozie to Airflow) |
[production] |
| 08:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 75%: Pooling', diff saved to https://phabricator.wikimedia.org/P46281 and previous config saved to /var/cache/conftool/dbconfig/20230411-081016-root.json |
[production] |
| 08:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 5%: Pooling', diff saved to https://phabricator.wikimedia.org/P46280 and previous config saved to /var/cache/conftool/dbconfig/20230411-080700-root.json |
[production] |
| 08:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 50%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46279 and previous config saved to /var/cache/conftool/dbconfig/20230411-080329-root.json |
[production] |
| 08:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P46278 and previous config saved to /var/cache/conftool/dbconfig/20230411-080057-root.json |
[production] |
| 07:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 50%: Pooling', diff saved to https://phabricator.wikimedia.org/P46277 and previous config saved to /var/cache/conftool/dbconfig/20230411-075511-root.json |
[production] |
| 07:54 |
<vgutierrez> |
restart haproxy on cp2033 - T334448 |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 4%: Pooling', diff saved to https://phabricator.wikimedia.org/P46276 and previous config saved to /var/cache/conftool/dbconfig/20230411-075155-root.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 25%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46275 and previous config saved to /var/cache/conftool/dbconfig/20230411-074824-root.json |
[production] |
| 07:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P46274 and previous config saved to /var/cache/conftool/dbconfig/20230411-074552-root.json |
[production] |
| 07:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 25%: Pooling', diff saved to https://phabricator.wikimedia.org/P46273 and previous config saved to /var/cache/conftool/dbconfig/20230411-074006-root.json |
[production] |
| 07:39 |
<jelto@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab2003.wikimedia.org with OS bullseye |
[production] |
| 07:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1103.eqiad.wmnet |
[production] |
| 07:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1103.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 07:37 |
<marostegui@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1103.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 3%: Pooling', diff saved to https://phabricator.wikimedia.org/P46272 and previous config saved to /var/cache/conftool/dbconfig/20230411-073651-root.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 10%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46271 and previous config saved to /var/cache/conftool/dbconfig/20230411-073319-root.json |
[production] |
| 07:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1103.eqiad.wmnet |
[production] |
| 07:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P46270 and previous config saved to /var/cache/conftool/dbconfig/20230411-073047-root.json |
[production] |
| 07:30 |
<dcausse> |
restarting blazegraph on wdqs1007 (stuck for 48hours) |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 10%: Pooling', diff saved to https://phabricator.wikimedia.org/P46269 and previous config saved to /var/cache/conftool/dbconfig/20230411-072501-root.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1211 (re)pooling @ 2%: Pooling', diff saved to https://phabricator.wikimedia.org/P46268 and previous config saved to /var/cache/conftool/dbconfig/20230411-072146-root.json |
[production] |
| 07:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1209 (re)pooling @ 5%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46267 and previous config saved to /var/cache/conftool/dbconfig/20230411-071815-root.json |
[production] |
| 07:18 |
<zabe@deploy2002> |
Finished scap: Backport for [[gerrit:906793|Add blkwiki to wgSitename (T334351)]] (duration: 08m 08s) |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1103 from dbctl T332293', diff saved to https://phabricator.wikimedia.org/P46266 and previous config saved to /var/cache/conftool/dbconfig/20230411-071647-marostegui.json |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P46265 and previous config saved to /var/cache/conftool/dbconfig/20230411-071542-root.json |
[production] |
| 07:11 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 393731 |
[production] |
| 07:11 |
<zabe@deploy2002> |
zabe and jhsoby: Backport for [[gerrit:906793|Add blkwiki to wgSitename (T334351)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 07:11 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 393731 |
[production] |
| 07:11 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 150279 |
[production] |
| 07:10 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 150279 |
[production] |
| 07:10 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 35467 |
[production] |
| 07:10 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 35467 |
[production] |
| 07:10 |
<zabe@deploy2002> |
Started scap: Backport for [[gerrit:906793|Add blkwiki to wgSitename (T334351)]] |
[production] |
| 07:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 5%: Pooling', diff saved to https://phabricator.wikimedia.org/P46264 and previous config saved to /var/cache/conftool/dbconfig/20230411-070956-root.json |
[production] |