|
2025-06-04
§
|
| 11:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2193 T395989', diff saved to https://phabricator.wikimedia.org/P77031 and previous config saved to /var/cache/conftool/dbconfig/20250604-110955-marostegui.json |
[production] |
| 10:35 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4007.ulsfo.wmnet |
[production] |
| 10:25 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4007.ulsfo.wmnet |
[production] |
| 10:23 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet |
[production] |
| 10:09 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet |
[production] |
| 10:00 |
<vgutierrez> |
depool lvs1013 before switching to katran - T395228 |
[production] |
| 09:42 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4005.ulsfo.wmnet |
[production] |
| 09:33 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4005.ulsfo.wmnet |
[production] |
| 09:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool pc1 T395983', diff saved to https://phabricator.wikimedia.org/P77006 and previous config saved to /var/cache/conftool/dbconfig/20250604-091921-marostegui.json |
[production] |
| 09:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es2044', diff saved to https://phabricator.wikimedia.org/P77002 and previous config saved to /var/cache/conftool/dbconfig/20250604-091041-marostegui.json |
[production] |
| 09:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es2041', diff saved to https://phabricator.wikimedia.org/P76999 and previous config saved to /var/cache/conftool/dbconfig/20250604-090226-marostegui.json |
[production] |
| 08:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2224 T395989', diff saved to https://phabricator.wikimedia.org/P76990 and previous config saved to /var/cache/conftool/dbconfig/20250604-081058-marostegui.json |
[production] |
| 07:01 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7001.magru.wmnet |
[production] |
| 06:57 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7001.magru.wmnet |
[production] |
| 06:04 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1035 T395982', diff saved to https://phabricator.wikimedia.org/P76972 and previous config saved to /var/cache/conftool/dbconfig/20250604-060413-marostegui.json |
[production] |
|
2025-06-02
§
|
| 17:05 |
<jasmine@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1026-1028].eqiad.wmnet |
[production] |
| 17:04 |
<jasmine@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1026-1028].eqiad.wmnet |
[production] |
| 16:22 |
<sukhe> |
sudo cumin -b1 -s60 'A:cp and not P{cp7001*}' "depool cdn && sleep 10 && run-puppet-agent --enable 'merging CR 1091330' && systemctl restart trafficserver.service && sleep 10 && pool cdn" |
[production] |
| 09:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es2039 T395647', diff saved to https://phabricator.wikimedia.org/P76798 and previous config saved to /var/cache/conftool/dbconfig/20250602-094402-marostegui.json |
[production] |
| 07:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1040 T395647', diff saved to https://phabricator.wikimedia.org/P76789 and previous config saved to /var/cache/conftool/dbconfig/20250602-070602-marostegui.json |
[production] |
| 05:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1169 T395663', diff saved to https://phabricator.wikimedia.org/P76781 and previous config saved to /var/cache/conftool/dbconfig/20250602-053905-marostegui.json |
[production] |
| 05:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es2036 T395771', diff saved to https://phabricator.wikimedia.org/P76779 and previous config saved to /var/cache/conftool/dbconfig/20250602-050150-marostegui.json |
[production] |
|
2025-05-28
§
|
| 16:18 |
<stevemunene@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker1119.eqiad.wmnet with reason: Repair data node volume failure |
[production] |
| 13:31 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new ncredir node for magru03 - jmm@cumin2002" |
[production] |
| 13:31 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new ncredir node for magru03 - jmm@cumin2002" |
[production] |
| 10:15 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db2242 - Depool db2242.codfw.wmnet to then clone it to db2187.codfw.wmnet - marostegui@cumin1002 |
[production] |
| 10:15 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.depool db2242 - Depool db2242.codfw.wmnet to then clone it to db2187.codfw.wmnet - marostegui@cumin1002 |
[production] |
| 09:56 |
<moritzm> |
installing node-serialize-javascript security updates |
[production] |
| 09:05 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es2035', diff saved to https://phabricator.wikimedia.org/P76551 and previous config saved to /var/cache/conftool/dbconfig/20250528-090528-marostegui.json |
[production] |
| 08:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1038 T394469', diff saved to https://phabricator.wikimedia.org/P76550 and previous config saved to /var/cache/conftool/dbconfig/20250528-083338-marostegui.json |
[production] |
|
2025-05-27
§
|
| 19:11 |
<inflatador> |
bking@cumin2002 depool cirrussearch106[0-6] T394350 |
[production] |
| 16:48 |
<jasmine@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1022-1025].eqiad.wmnet |
[production] |
| 16:43 |
<jasmine@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1022-1025].eqiad.wmnet |
[production] |
| 16:21 |
<klausman@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet |
[production] |
| 16:21 |
<klausman@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet |
[production] |
| 15:23 |
<klausman@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet |
[production] |
| 15:13 |
<klausman@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet |
[production] |
| 15:12 |
<klausman@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet |
[production] |
| 15:12 |
<klausman@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet |
[production] |
| 14:58 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet |
[production] |
| 14:58 |
<elukey@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet |
[production] |
| 13:52 |
<klausman@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet |
[production] |