|
2022-11-07
ยง
|
| 11:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38235 and previous config saved to /var/cache/conftool/dbconfig/20221107-111616-marostegui.json |
[production] |
| 11:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P38234 and previous config saved to /var/cache/conftool/dbconfig/20221107-111615-ladsgroup.json |
[production] |
| 11:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38233 and previous config saved to /var/cache/conftool/dbconfig/20221107-111351-ladsgroup.json |
[production] |
| 11:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T321123)', diff saved to https://phabricator.wikimedia.org/P38232 and previous config saved to /var/cache/conftool/dbconfig/20221107-111156-marostegui.json |
[production] |
| 11:06 |
<arturo> |
running homer on cr-eqiad/cr-codfw for https://gerrit.wikimedia.org/r/c/operations/homer/public/+/853374 (T321220, T309407) |
[production] |
| 11:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1002.eqiad.wmnet to drbd |
[production] |
| 11:04 |
<_joe_> |
manually started dump_cloud_ip_ranges.service |
[production] |
| 11:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38231 and previous config saved to /var/cache/conftool/dbconfig/20221107-110215-root.json |
[production] |
| 11:01 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: sync |
[production] |
| 11:01 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: sync |
[production] |
| 11:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38230 and previous config saved to /var/cache/conftool/dbconfig/20221107-110110-marostegui.json |
[production] |
| 11:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38229 and previous config saved to /var/cache/conftool/dbconfig/20221107-110109-ladsgroup.json |
[production] |
| 10:59 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: apply |
[production] |
| 10:59 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: apply |
[production] |
| 10:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P38228 and previous config saved to /var/cache/conftool/dbconfig/20221107-105844-ladsgroup.json |
[production] |
| 10:57 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: sync |
[production] |
| 10:57 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: sync |
[production] |
| 10:56 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1002.eqiad.wmnet to drbd |
[production] |
| 10:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1157 (T318955)', diff saved to https://phabricator.wikimedia.org/P38227 and previous config saved to /var/cache/conftool/dbconfig/20221107-105015-ladsgroup.json |
[production] |
| 10:50 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:49 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:49 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 10:49 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 10:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38226 and previous config saved to /var/cache/conftool/dbconfig/20221107-104710-root.json |
[production] |
| 10:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P38225 and previous config saved to /var/cache/conftool/dbconfig/20221107-104603-marostegui.json |
[production] |
| 10:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38224 and previous config saved to /var/cache/conftool/dbconfig/20221107-104338-ladsgroup.json |
[production] |
| 10:41 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:41 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112 (T318955)', diff saved to https://phabricator.wikimedia.org/P38223 and previous config saved to /var/cache/conftool/dbconfig/20221107-104101-ladsgroup.json |
[production] |
| 10:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2109 (T318955)', diff saved to https://phabricator.wikimedia.org/P38222 and previous config saved to /var/cache/conftool/dbconfig/20221107-103622-ladsgroup.json |
[production] |
| 10:36 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 10:35 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 10:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105 (T318955)', diff saved to https://phabricator.wikimedia.org/P38221 and previous config saved to /var/cache/conftool/dbconfig/20221107-103549-ladsgroup.json |
[production] |
| 10:32 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 5398 |
[production] |
| 10:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P38220 and previous config saved to /var/cache/conftool/dbconfig/20221107-103205-root.json |
[production] |
| 10:31 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 5398 |
[production] |
| 10:31 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30103 |
[production] |
| 10:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T321130)', diff saved to https://phabricator.wikimedia.org/P38219 and previous config saved to /var/cache/conftool/dbconfig/20221107-103056-marostegui.json |
[production] |
| 10:30 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 30103 |
[production] |
| 10:29 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 399338 |
[production] |
| 10:29 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 399338 |
[production] |
| 10:28 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 59796 |
[production] |
| 10:28 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 59796 |
[production] |
| 10:27 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4817 |
[production] |
| 10:26 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 4817 |
[production] |
| 10:26 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 35598 |
[production] |
| 10:26 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 35598 |
[production] |
| 10:26 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 30058 |
[production] |
| 10:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P38218 and previous config saved to /var/cache/conftool/dbconfig/20221107-102555-ladsgroup.json |
[production] |