|
2025-08-05
ยง
|
| 23:59 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P80871 and previous config saved to /var/cache/conftool/dbconfig/20250805-235905-fceratto.json |
[production] |
| 23:55 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T386098, transfer newly-reloaded data) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 23:43 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1226 (T399728)', diff saved to https://phabricator.wikimedia.org/P80870 and previous config saved to /var/cache/conftool/dbconfig/20250805-234358-fceratto.json |
[production] |
| 23:39 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1226 (T399728)', diff saved to https://phabricator.wikimedia.org/P80869 and previous config saved to /var/cache/conftool/dbconfig/20250805-233907-fceratto.json |
[production] |
| 23:39 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1226.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:38 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214 (T399728)', diff saved to https://phabricator.wikimedia.org/P80868 and previous config saved to /var/cache/conftool/dbconfig/20250805-233843-fceratto.json |
[production] |
| 23:23 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P80867 and previous config saved to /var/cache/conftool/dbconfig/20250805-232336-fceratto.json |
[production] |
| 23:08 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P80866 and previous config saved to /var/cache/conftool/dbconfig/20250805-230828-fceratto.json |
[production] |
| 22:55 |
<bking@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T386098, transfer newly-reloaded data) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 22:53 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214 (T399728)', diff saved to https://phabricator.wikimedia.org/P80865 and previous config saved to /var/cache/conftool/dbconfig/20250805-225320-fceratto.json |
[production] |
| 22:52 |
<bking@cumin2002> |
conftool action : set/pooled=true; selector: dnsdisc=wdqs-scholarly,name=eqiad |
[production] |
| 22:48 |
<sfaci@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply |
[production] |
| 22:48 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1214 (T399728)', diff saved to https://phabricator.wikimedia.org/P80864 and previous config saved to /var/cache/conftool/dbconfig/20250805-224824-fceratto.json |
[production] |
| 22:48 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1214.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:48 |
<sfaci@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply |
[production] |
| 22:48 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209 (T399728)', diff saved to https://phabricator.wikimedia.org/P80863 and previous config saved to /var/cache/conftool/dbconfig/20250805-224801-fceratto.json |
[production] |
| 22:32 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P80862 and previous config saved to /var/cache/conftool/dbconfig/20250805-223253-fceratto.json |
[production] |
| 22:18 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1007.eqiad.wmnet with OS bookworm |
[production] |
| 22:18 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" |
[production] |
| 22:17 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P80861 and previous config saved to /var/cache/conftool/dbconfig/20250805-221746-fceratto.json |
[production] |
| 22:17 |
<jclark@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" |
[production] |
| 22:05 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T386098, transfer newly-reloaded data) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1024.eqiad.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 22:02 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209 (T399728)', diff saved to https://phabricator.wikimedia.org/P80860 and previous config saved to /var/cache/conftool/dbconfig/20250805-220238-fceratto.json |
[production] |
| 21:59 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1007.eqiad.wmnet with reason: host reimage |
[production] |
| 21:57 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1209 (T399728)', diff saved to https://phabricator.wikimedia.org/P80859 and previous config saved to /var/cache/conftool/dbconfig/20250805-215738-fceratto.json |
[production] |
| 21:57 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1209.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:57 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203 (T399728)', diff saved to https://phabricator.wikimedia.org/P80858 and previous config saved to /var/cache/conftool/dbconfig/20250805-215715-fceratto.json |
[production] |
| 21:55 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1007.eqiad.wmnet with reason: host reimage |
[production] |
| 21:46 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T386098, transfer newly-reloaded data) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2010.codfw.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 21:42 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P80857 and previous config saved to /var/cache/conftool/dbconfig/20250805-214208-fceratto.json |
[production] |
| 21:41 |
<jgleeson> |
SmashPig upgraded from a7e897ec to 83293ee1 |
[production] |
| 21:37 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.reimage for host dbprov1007.eqiad.wmnet with OS bookworm |
[production] |
| 21:31 |
<dani@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 21:31 |
<dani@deploy1003> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 21:31 |
<dani@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 21:31 |
<dani@deploy1003> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 21:31 |
<dani@deploy1003> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 21:31 |
<dani@deploy1003> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 21:31 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1007.eqiad.wmnet with OS bookworm |
[production] |
| 21:27 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P80856 and previous config saved to /var/cache/conftool/dbconfig/20250805-212701-fceratto.json |
[production] |
| 21:25 |
<brett@dns1004> |
END - running authdns-update |
[production] |
| 21:25 |
<dani@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 21:25 |
<dani@deploy1003> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 21:24 |
<dani@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 21:24 |
<dani@deploy1003> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 21:24 |
<dani@deploy1003> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 21:24 |
<dani@deploy1003> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 21:23 |
<bking@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=wdqs-scholarly,name=eqiad |
[production] |
| 21:22 |
<brett@dns1004> |
START - running authdns-update |
[production] |
| 21:18 |
<bking@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T386098, transfer newly-reloaded data) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1024.eqiad.wmnet w/ force delete existing files, repooling both afterwards |
[production] |