|
2024-07-11
ยง
|
| 02:55 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T367781)', diff saved to https://phabricator.wikimedia.org/P66232 and previous config saved to /var/cache/conftool/dbconfig/20240711-025537-arnaudb.json |
[production] |
| 02:48 |
<eileen> |
civicrm upgraded from a17496a2 to 2d1a0aad |
[production] |
| 02:45 |
<mutante> |
stewards2001 - sudo mv /srv/repos/users-db /root/ - run puppet and let it recreate the usersdb repo - this time pulling from gitlab - T369780 T369430 |
[production] |
| 02:40 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P66231 and previous config saved to /var/cache/conftool/dbconfig/20240711-024030-arnaudb.json |
[production] |
| 02:25 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P66230 and previous config saved to /var/cache/conftool/dbconfig/20240711-022522-arnaudb.json |
[production] |
| 02:15 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 02:15 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate True, for hosts list: ['cloudvirt1060'] |
[cloudvirt-canary] |
| 02:14 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1060.eqiad.wmnet with OS bookworm |
[production] |
| 02:14 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate True, for hosts list: ['cloudvirt1060'] |
[cloudvirt-canary] |
| 02:12 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 02:10 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T367781)', diff saved to https://phabricator.wikimedia.org/P66229 and previous config saved to /var/cache/conftool/dbconfig/20240711-021015-arnaudb.json |
[production] |
| 02:08 |
<eileen> |
civicrm upgraded from a03085ff to 1e2fcba3 |
[production] |
| 02:08 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2164 (T367781)', diff saved to https://phabricator.wikimedia.org/P66228 and previous config saved to /var/cache/conftool/dbconfig/20240711-020805-arnaudb.json |
[production] |
| 02:07 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 02:07 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 02:07 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 02:07 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 02:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T367781)', diff saved to https://phabricator.wikimedia.org/P66227 and previous config saved to /var/cache/conftool/dbconfig/20240711-020738-arnaudb.json |
[production] |
| 02:07 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 02:07 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 02:02 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 02:02 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 02:02 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 02:02 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:54 |
<eileen> |
config revision changed from 840e6b90 to e02c3a85 |
[production] |
| 01:53 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:53 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:52 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P66226 and previous config saved to /var/cache/conftool/dbconfig/20240711-015231-arnaudb.json |
[production] |
| 01:51 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:51 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:50 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:50 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:48 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:48 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:47 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:47 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:46 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1060.eqiad.wmnet with reason: host reimage |
[production] |
| 01:44 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:44 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:43 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:43 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:43 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:43 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:43 |
<andrew@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1060.eqiad.wmnet with reason: host reimage |
[production] |
| 01:37 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P66225 and previous config saved to /var/cache/conftool/dbconfig/20240711-013723-arnaudb.json |
[production] |
| 01:36 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:36 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet' |
[admin] |
| 01:27 |
<andrew@cumin1002> |
START - Cookbook sre.hosts.reimage for host cloudvirt1060.eqiad.wmnet with OS bookworm |
[production] |
| 01:22 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T367781)', diff saved to https://phabricator.wikimedia.org/P66224 and previous config saved to /var/cache/conftool/dbconfig/20240711-012216-arnaudb.json |
[production] |
| 01:21 |
<mutante> |
gerrit-replica.wikimedia.org (gerrit2002) - switched firewall provider from iptables to nftables - all seems fine to me but just in case: gerrit:1053068 can be reverted to go back |
[production] |