|
2022-11-29
ยง
|
| 10:26 |
<elukey> |
restart kube-apiserver on ml-serve-ctrl* to clear out some knative controller issue |
[production] |
| 10:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2120 (T323907)', diff saved to https://phabricator.wikimedia.org/P41666 and previous config saved to /var/cache/conftool/dbconfig/20221129-102345-ladsgroup.json |
[production] |
| 10:19 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T323907)', diff saved to https://phabricator.wikimedia.org/P41665 and previous config saved to /var/cache/conftool/dbconfig/20221129-101958-ladsgroup.json |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P41664 and previous config saved to /var/cache/conftool/dbconfig/20221129-101754-marostegui.json |
[production] |
| 10:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41663 and previous config saved to /var/cache/conftool/dbconfig/20221129-101554-root.json |
[production] |
| 10:15 |
<moritzm> |
upgrading puppetdb2003 to bookworm T321783 |
[production] |
| 10:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P41662 and previous config saved to /var/cache/conftool/dbconfig/20221129-101225-ladsgroup.json |
[production] |
| 10:07 |
<jynus> |
add temporary grants to scholarships for backups on db1117, db2160 T243037 |
[production] |
| 10:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1194 (T323907)', diff saved to https://phabricator.wikimedia.org/P41661 and previous config saved to /var/cache/conftool/dbconfig/20221129-100319-ladsgroup.json |
[production] |
| 10:03 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:03 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T323907)', diff saved to https://phabricator.wikimedia.org/P41660 and previous config saved to /var/cache/conftool/dbconfig/20221129-100258-ladsgroup.json |
[production] |
| 10:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1118 (T321126)', diff saved to https://phabricator.wikimedia.org/P41659 and previous config saved to /var/cache/conftool/dbconfig/20221129-100248-marostegui.json |
[production] |
| 10:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41658 and previous config saved to /var/cache/conftool/dbconfig/20221129-100049-root.json |
[production] |
| 10:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1118 (T321126)', diff saved to https://phabricator.wikimedia.org/P41657 and previous config saved to /var/cache/conftool/dbconfig/20221129-100025-marostegui.json |
[production] |
| 10:00 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1118.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:59 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1118.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321126)', diff saved to https://phabricator.wikimedia.org/P41656 and previous config saved to /var/cache/conftool/dbconfig/20221129-095931-marostegui.json |
[production] |
| 09:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P41655 and previous config saved to /var/cache/conftool/dbconfig/20221129-095718-ladsgroup.json |
[production] |
| 09:56 |
<moritzm> |
installing curl security updates |
[production] |
| 09:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2120 (T323907)', diff saved to https://phabricator.wikimedia.org/P41654 and previous config saved to /var/cache/conftool/dbconfig/20221129-094818-ladsgroup.json |
[production] |
| 09:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance |
[production] |
| 09:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance |
[production] |
| 09:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108 (T323907)', diff saved to https://phabricator.wikimedia.org/P41653 and previous config saved to /var/cache/conftool/dbconfig/20221129-094757-ladsgroup.json |
[production] |
| 09:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P41652 and previous config saved to /var/cache/conftool/dbconfig/20221129-094745-ladsgroup.json |
[production] |
| 09:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41651 and previous config saved to /var/cache/conftool/dbconfig/20221129-094544-root.json |
[production] |
| 09:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P41650 and previous config saved to /var/cache/conftool/dbconfig/20221129-094424-marostegui.json |
[production] |
| 09:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P41649 and previous config saved to /var/cache/conftool/dbconfig/20221129-094212-ladsgroup.json |
[production] |
| 09:34 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2149 (T322618)', diff saved to https://phabricator.wikimedia.org/P41648 and previous config saved to /var/cache/conftool/dbconfig/20221129-093420-ladsgroup.json |
[production] |
| 09:34 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance |
[production] |
| 09:34 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance |
[production] |
| 09:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P41647 and previous config saved to /var/cache/conftool/dbconfig/20221129-093250-ladsgroup.json |
[production] |
| 09:32 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P41646 and previous config saved to /var/cache/conftool/dbconfig/20221129-093239-ladsgroup.json |
[production] |
| 09:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41645 and previous config saved to /var/cache/conftool/dbconfig/20221129-093039-root.json |
[production] |
| 09:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P41644 and previous config saved to /var/cache/conftool/dbconfig/20221129-092918-marostegui.json |
[production] |
| 09:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 09:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 09:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T322618)', diff saved to https://phabricator.wikimedia.org/P41643 and previous config saved to /var/cache/conftool/dbconfig/20221129-092822-ladsgroup.json |
[production] |
| 09:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P41642 and previous config saved to /var/cache/conftool/dbconfig/20221129-091744-ladsgroup.json |
[production] |
| 09:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1191 (T323907)', diff saved to https://phabricator.wikimedia.org/P41641 and previous config saved to /var/cache/conftool/dbconfig/20221129-091732-ladsgroup.json |
[production] |
| 09:17 |
<moritzm> |
update component/puppetdb7 to puppetdb 7.11.2-3 (fixing Postgres 15 compat) T321783 |
[production] |
| 09:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P41640 and previous config saved to /var/cache/conftool/dbconfig/20221129-091534-root.json |
[production] |
| 09:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321126)', diff saved to https://phabricator.wikimedia.org/P41639 and previous config saved to /var/cache/conftool/dbconfig/20221129-091412-marostegui.json |
[production] |
| 09:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P41638 and previous config saved to /var/cache/conftool/dbconfig/20221129-091315-ladsgroup.json |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2145', diff saved to https://phabricator.wikimedia.org/P41637 and previous config saved to /var/cache/conftool/dbconfig/20221129-091224-marostegui.json |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1107 (T321126)', diff saved to https://phabricator.wikimedia.org/P41636 and previous config saved to /var/cache/conftool/dbconfig/20221129-091149-marostegui.json |
[production] |
| 09:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1107.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:11 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1107.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321126)', diff saved to https://phabricator.wikimedia.org/P41635 and previous config saved to /var/cache/conftool/dbconfig/20221129-091117-marostegui.json |
[production] |
| 09:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108 (T323907)', diff saved to https://phabricator.wikimedia.org/P41634 and previous config saved to /var/cache/conftool/dbconfig/20221129-090237-ladsgroup.json |
[production] |