|
2024-02-29
ยง
|
| 10:24 |
<brouberol@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-tool1005.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin1002" |
[production] |
| 10:24 |
<claime> |
Cordoning kubernetes2023.codfw.wmnet for vlan change cookbook tests - T350152 |
[production] |
| 10:21 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1196 (T357189)', diff saved to https://phabricator.wikimedia.org/P58223 and previous config saved to /var/cache/conftool/dbconfig/20240229-102143-arnaudb.json |
[production] |
| 10:21 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:21 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:21 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:21 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:21 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186 (T357189)', diff saved to https://phabricator.wikimedia.org/P58222 and previous config saved to /var/cache/conftool/dbconfig/20240229-102102-arnaudb.json |
[production] |
| 10:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 75%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P58221 and previous config saved to /var/cache/conftool/dbconfig/20240229-101926-arnaudb.json |
[production] |
| 10:17 |
<joal@deploy2002> |
Finished deploy [analytics/refinery@6e8f25b] (hadoop-test): Additional analytics weekly train - TEST [analytics/refinery@6e8f25b3] (duration: 03m 41s) |
[production] |
| 10:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2034 (re)pooling @ 50%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P58220 and previous config saved to /var/cache/conftool/dbconfig/20240229-101427-root.json |
[production] |
| 10:13 |
<joal@deploy2002> |
Started deploy [analytics/refinery@6e8f25b] (hadoop-test): Additional analytics weekly train - TEST [analytics/refinery@6e8f25b3] |
[production] |
| 10:12 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P58219 and previous config saved to /var/cache/conftool/dbconfig/20240229-101214-arnaudb.json |
[production] |
| 10:11 |
<joal@deploy2002> |
Finished deploy [analytics/refinery@6e8f25b] (thin): Additional analytics weekly train - THIN [analytics/refinery@6e8f25b3] (duration: 00m 05s) |
[production] |
| 10:11 |
<joal@deploy2002> |
Started deploy [analytics/refinery@6e8f25b] (thin): Additional analytics weekly train - THIN [analytics/refinery@6e8f25b3] |
[production] |
| 10:11 |
<joal@deploy2002> |
Finished deploy [analytics/refinery@6e8f25b]: Additional analytics weekly train [analytics/refinery@6e8f25b3] (duration: 11m 39s) |
[production] |
| 10:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P58218 and previous config saved to /var/cache/conftool/dbconfig/20240229-100556-arnaudb.json |
[production] |
| 10:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P58217 and previous config saved to /var/cache/conftool/dbconfig/20240229-100421-arnaudb.json |
[production] |
| 09:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2034 (re)pooling @ 25%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P58216 and previous config saved to /var/cache/conftool/dbconfig/20240229-095923-root.json |
[production] |
| 09:59 |
<joal@deploy2002> |
Started deploy [analytics/refinery@6e8f25b]: Additional analytics weekly train [analytics/refinery@6e8f25b3] |
[production] |
| 09:59 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'T356240 ', diff saved to https://phabricator.wikimedia.org/P58215 and previous config saved to /var/cache/conftool/dbconfig/20240229-095918-arnaudb.json |
[production] |
| 09:58 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2117.codfw.wmnet with reason: Silence for maintenance T356240 |
[production] |
| 09:57 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on db2117.codfw.wmnet with reason: Silence for maintenance T356240 |
[production] |
| 09:57 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 25%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P58214 and previous config saved to /var/cache/conftool/dbconfig/20240229-095709-arnaudb.json |
[production] |
| 09:55 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1224.eqiad.wmnet |
[production] |
| 09:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 100%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P58213 and previous config saved to /var/cache/conftool/dbconfig/20240229-095425-root.json |
[production] |
| 09:51 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1224.eqiad.wmnet |
[production] |
| 09:51 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db1224.eqiad.wmnet with reason: Silence for maintenance T356240 |
[production] |
| 09:51 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on db1224.eqiad.wmnet with reason: Silence for maintenance T356240 |
[production] |
| 09:50 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P58212 and previous config saved to /var/cache/conftool/dbconfig/20240229-095049-arnaudb.json |
[production] |
| 09:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'T356240 reboot', diff saved to https://phabricator.wikimedia.org/P58211 and previous config saved to /var/cache/conftool/dbconfig/20240229-094945-arnaudb.json |
[production] |
| 09:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 25%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P58210 and previous config saved to /var/cache/conftool/dbconfig/20240229-094915-arnaudb.json |
[production] |
| 09:44 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1231 (T352010)', diff saved to https://phabricator.wikimedia.org/P58209 and previous config saved to /var/cache/conftool/dbconfig/20240229-094429-ladsgroup.json |
[production] |
| 09:44 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2034 (re)pooling @ 10%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P58208 and previous config saved to /var/cache/conftool/dbconfig/20240229-094418-root.json |
[production] |
| 09:44 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1201.eqiad.wmnet |
[production] |
| 09:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 75%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P58207 and previous config saved to /var/cache/conftool/dbconfig/20240229-093921-root.json |
[production] |
| 09:36 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1201.eqiad.wmnet |
[production] |
| 09:35 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186 (T357189)', diff saved to https://phabricator.wikimedia.org/P58206 and previous config saved to /var/cache/conftool/dbconfig/20240229-093543-arnaudb.json |
[production] |
| 09:34 |
<brouberol@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1186 (T357189)', diff saved to https://phabricator.wikimedia.org/P58205 and previous config saved to /var/cache/conftool/dbconfig/20240229-093025-arnaudb.json |
[production] |
| 09:30 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:30 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T357189)', diff saved to https://phabricator.wikimedia.org/P58204 and previous config saved to /var/cache/conftool/dbconfig/20240229-093003-arnaudb.json |
[production] |
| 09:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote back es2034 to es3 codfw master T358180', diff saved to https://phabricator.wikimedia.org/P58203 and previous config saved to /var/cache/conftool/dbconfig/20240229-092929-marostegui.json |
[production] |
| 09:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2034 (re)pooling @ 5%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P58202 and previous config saved to /var/cache/conftool/dbconfig/20240229-092913-root.json |
[production] |
| 09:28 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'depooling for maintenance - reboot', diff saved to https://phabricator.wikimedia.org/P58201 and previous config saved to /var/cache/conftool/dbconfig/20240229-092853-arnaudb.json |
[production] |
| 09:26 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2034.codfw.wmnet with OS bookworm |
[production] |
| 09:26 |
<brouberol@cumin1002> |
START - Cookbook sre.hosts.decommission for hosts an-tool1005.eqiad.wmnet |
[production] |