|
2023-11-16
§
|
| 05:36 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T348183)', diff saved to https://phabricator.wikimedia.org/P53498 and previous config saved to /var/cache/conftool/dbconfig/20231116-053554-arnaudb.json |
[production] |
| 05:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P53497 and previous config saved to /var/cache/conftool/dbconfig/20231116-052048-arnaudb.json |
[production] |
| 05:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P53496 and previous config saved to /var/cache/conftool/dbconfig/20231116-050542-arnaudb.json |
[production] |
| 04:57 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2001-dev.codfw.wmnet with OS bookworm |
[production] |
| 04:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T348183)', diff saved to https://phabricator.wikimedia.org/P53495 and previous config saved to /var/cache/conftool/dbconfig/20231116-045035-arnaudb.json |
[production] |
| 04:38 |
<cstone> |
payments-wiki upgraded from 6affb60a to eae2f35e |
[production] |
| 04:30 |
<cstone> |
payments-wiki upgraded from 084370bb to 6affb60a |
[production] |
| 04:24 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudelastic1007.wikimedia.org with OS bullseye |
[production] |
| 03:44 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2001-dev.codfw.wmnet with reason: host reimage |
[production] |
| 03:40 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2001-dev.codfw.wmnet with reason: host reimage |
[production] |
| 03:40 |
<ejegg> |
fundraising civicrm upgraded from 6e53198c to 32679ea3 |
[production] |
| 03:19 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudcontrol2001-dev.codfw.wmnet with OS bookworm |
[production] |
| 01:53 |
<cstone> |
payments-wiki upgraded from b4465e23 to 084370bb |
[production] |
| 01:34 |
<eileen> |
civicrm upgraded from ec6992e0 to 6e53198c |
[production] |
| 00:27 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1008.wikimedia.org with OS bullseye |
[production] |
|
2023-11-15
§
|
| 23:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db1142 (T348183)', diff saved to https://phabricator.wikimedia.org/P53494 and previous config saved to /var/cache/conftool/dbconfig/20231115-235044-arnaudb.json |
[production] |
| 23:50 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:50 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T348183)', diff saved to https://phabricator.wikimedia.org/P53493 and previous config saved to /var/cache/conftool/dbconfig/20231115-235023-arnaudb.json |
[production] |
| 23:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P53492 and previous config saved to /var/cache/conftool/dbconfig/20231115-233516-arnaudb.json |
[production] |
| 23:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P53491 and previous config saved to /var/cache/conftool/dbconfig/20231115-232010-arnaudb.json |
[production] |
| 23:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T348183)', diff saved to https://phabricator.wikimedia.org/P53490 and previous config saved to /var/cache/conftool/dbconfig/20231115-230504-arnaudb.json |
[production] |
| 23:04 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudelastic1008.wikimedia.org with OS bullseye |
[production] |
| 22:59 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for cloudelastic1007.wikimedia.org: Renew puppet certificate - bking@cumin2002 |
[production] |
| 22:58 |
<bking@cumin2002> |
START - Cookbook sre.puppet.renew-cert for cloudelastic1007.wikimedia.org: Renew puppet certificate - bking@cumin2002 |
[production] |
| 22:57 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for cloudelastic1008.wikimedia.org: Renew puppet certificate - bking@cumin2002 |
[production] |
| 22:57 |
<bking@cumin2002> |
START - Cookbook sre.puppet.renew-cert for cloudelastic1008.wikimedia.org: Renew puppet certificate - bking@cumin2002 |
[production] |
| 22:41 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cloudelastic[1007-1010].wikimedia.org with reason: new cloudelastic hosts TT351354 |
[production] |
| 22:41 |
<ryankemper@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cloudelastic[1007-1010].wikimedia.org with reason: new cloudelastic hosts TT351354 |
[production] |
| 22:20 |
<ryankemper> |
T351354 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/974693; running puppet on hosts |
[production] |
| 19:39 |
<topranks> |
re-enabling puppet on DNS hosts to adjust TTL setting in BIRD (T350488) |
[production] |
| 19:37 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1010.wikimedia.org with OS bullseye |
[production] |
| 19:36 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1009.wikimedia.org with OS bullseye |
[production] |
| 19:34 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudelastic1008.wikimedia.org with OS bullseye |
[production] |
| 19:23 |
<jhuneidi@deploy2002> |
Synchronized php: group1 wikis to 1.42.0-wmf.5 refs T350081 (duration: 05m 52s) |
[production] |
| 19:17 |
<jhuneidi@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.5 refs T350081 |
[production] |
| 19:15 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: aphlict |
[production] |
| 19:10 |
<topranks> |
merging patch to remove TTL restriction on Bird Anycast BGP peerings (T350488) |
[production] |
| 19:09 |
<dzahn@cumin1001> |
START - Cookbook sre.puppet.migrate-role for role: aphlict |
[production] |
| 19:07 |
<taavi@cumin1001> |
END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudlb2001-dev.codfw.wmnet |
[production] |
| 19:07 |
<mutante> |
aphlict2001 - restart aphlict service after puppet 7 upgrade |
[production] |
| 19:05 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::codfw1dev::virt_ceph |
[production] |
| 19:01 |
<taavi@cumin1001> |
START - Cookbook sre.puppet.migrate-host for host cloudlb2001-dev.codfw.wmnet |
[production] |
| 19:00 |
<taavi@cumin1001> |
END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudgw2003-dev.codfw.wmnet |
[production] |
| 18:59 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::codfw1dev::services |
[production] |
| 18:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host aphlict2001.codfw.wmnet |
[production] |
| 18:59 |
<jbond@cumin1001> |
START - Cookbook sre.puppet.migrate-role for role: wmcs::openstack::codfw1dev::virt_ceph |
[production] |
| 18:58 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.puppet.migrate-role (exit_code=99) for role: wmcs::openstack::codfw1dev::virt_ceph |
[production] |
| 18:56 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS bullseye |
[production] |