|
2023-02-08
ยง
|
| 13:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P43813 and previous config saved to /var/cache/conftool/dbconfig/20230208-135331-marostegui.json |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2104 (T328817)', diff saved to https://phabricator.wikimedia.org/P43812 and previous config saved to /var/cache/conftool/dbconfig/20230208-135216-marostegui.json |
[production] |
| 13:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2104 (T328817)', diff saved to https://phabricator.wikimedia.org/P43811 and previous config saved to /var/cache/conftool/dbconfig/20230208-134950-marostegui.json |
[production] |
| 13:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2104.codfw.wmnet with reason: Maintenance |
[production] |
| 13:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2104.codfw.wmnet with reason: Maintenance |
[production] |
| 13:45 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 13:44 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 13:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P43810 and previous config saved to /var/cache/conftool/dbconfig/20230208-133825-marostegui.json |
[production] |
| 13:33 |
<jbond> |
(correction) send puppet.esams.wmnet to eqiad and puppet.esqin.wmnet to codfw |
[production] |
| 13:33 |
<jbond> |
send puppet.esams.wmnet to eqiad and puppet.esams.wmnet to codfw |
[production] |
| 13:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T328817)', diff saved to https://phabricator.wikimedia.org/P43809 and previous config saved to /var/cache/conftool/dbconfig/20230208-132318-marostegui.json |
[production] |
| 13:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1187 (T328817)', diff saved to https://phabricator.wikimedia.org/P43808 and previous config saved to /var/cache/conftool/dbconfig/20230208-131409-marostegui.json |
[production] |
| 13:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T328817)', diff saved to https://phabricator.wikimedia.org/P43807 and previous config saved to /var/cache/conftool/dbconfig/20230208-131348-marostegui.json |
[production] |
| 13:03 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1002.eqiad.wmnet with OS bullseye |
[production] |
| 13:00 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13150 |
[production] |
| 12:59 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 13150 |
[production] |
| 12:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P43805 and previous config saved to /var/cache/conftool/dbconfig/20230208-125841-marostegui.json |
[production] |
| 12:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P43804 and previous config saved to /var/cache/conftool/dbconfig/20230208-124335-marostegui.json |
[production] |
| 12:37 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1096.eqiad.wmnet |
[production] |
| 12:37 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:37 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1096.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 12:36 |
<marostegui@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1096.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" |
[production] |
| 12:34 |
<marostegui@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:29 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1096.eqiad.wmnet |
[production] |
| 12:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T328817)', diff saved to https://phabricator.wikimedia.org/P43803 and previous config saved to /var/cache/conftool/dbconfig/20230208-122829-marostegui.json |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1180 (T328817)', diff saved to https://phabricator.wikimedia.org/P43802 and previous config saved to /var/cache/conftool/dbconfig/20230208-122620-marostegui.json |
[production] |
| 12:26 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T328817)', diff saved to https://phabricator.wikimedia.org/P43801 and previous config saved to /var/cache/conftool/dbconfig/20230208-122559-marostegui.json |
[production] |
| 12:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling restart_daemons on A:ldap-replicas |
[production] |
| 12:19 |
<jmm@cumin2002> |
START - Cookbook sre.ldap.roll-restart-reboot-replica rolling restart_daemons on A:ldap-replicas |
[production] |
| 12:18 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage |
[production] |
| 12:15 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage |
[production] |
| 12:13 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kafka-stretch2002.codfw.wmnet |
[production] |
| 12:13 |
<jmm@cumin2002> |
START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kafka-stretch2002.codfw.wmnet |
[production] |
| 12:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P43800 and previous config saved to /var/cache/conftool/dbconfig/20230208-121053-marostegui.json |
[production] |
| 12:03 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.reimage for host gitlab-runner1002.eqiad.wmnet with OS bullseye |
[production] |
| 11:59 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: Attempting to move some GPUs |
[production] |
| 11:59 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: Attempting to move some GPUs |
[production] |
| 11:58 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-worker1097.eqiad.wmnet with reason: Attempting to move some GPUs |
[production] |
| 11:57 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on an-worker1097.eqiad.wmnet with reason: Attempting to move some GPUs |
[production] |
| 11:57 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-worker1096.eqiad.wmnet with reason: Attempting to move some GPUs |
[production] |
| 11:57 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on an-worker1096.eqiad.wmnet with reason: Attempting to move some GPUs |
[production] |
| 11:56 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: moss-be1001.eqiad.wmnet |
[production] |
| 11:56 |
<jmm@cumin2002> |
START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: moss-be1001.eqiad.wmnet |
[production] |
| 11:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P43799 and previous config saved to /var/cache/conftool/dbconfig/20230208-115546-marostegui.json |
[production] |
| 11:53 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: flowspec1001.eqiad.wmnet |
[production] |
| 11:53 |
<jmm@cumin2002> |
START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: flowspec1001.eqiad.wmnet |
[production] |