|
2022-11-29
ยง
|
| 13:30 |
<pt1979@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for db120[4-5] - pt1979@cumin2002" |
[production] |
| 13:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P41718 and previous config saved to /var/cache/conftool/dbconfig/20221129-133005-marostegui.json |
[production] |
| 13:28 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P41717 and previous config saved to /var/cache/conftool/dbconfig/20221129-132513-ladsgroup.json |
[production] |
| 13:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134 (T321126)', diff saved to https://phabricator.wikimedia.org/P41715 and previous config saved to /var/cache/conftool/dbconfig/20221129-131459-marostegui.json |
[production] |
| 13:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1134 (T321126)', diff saved to https://phabricator.wikimedia.org/P41714 and previous config saved to /var/cache/conftool/dbconfig/20221129-131238-marostegui.json |
[production] |
| 13:12 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1134.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1134.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321126)', diff saved to https://phabricator.wikimedia.org/P41713 and previous config saved to /var/cache/conftool/dbconfig/20221129-131216-marostegui.json |
[production] |
| 13:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2122 (T323907)', diff saved to https://phabricator.wikimedia.org/P41712 and previous config saved to /var/cache/conftool/dbconfig/20221129-131006-ladsgroup.json |
[production] |
| 13:00 |
<moritzm> |
installing glibc security updates on buster |
[production] |
| 12:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P41711 and previous config saved to /var/cache/conftool/dbconfig/20221129-125710-marostegui.json |
[production] |
| 12:51 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:51 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T323907)', diff saved to https://phabricator.wikimedia.org/P41710 and previous config saved to /var/cache/conftool/dbconfig/20221129-125121-ladsgroup.json |
[production] |
| 12:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P41709 and previous config saved to /var/cache/conftool/dbconfig/20221129-124203-marostegui.json |
[production] |
| 12:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P41708 and previous config saved to /var/cache/conftool/dbconfig/20221129-123614-ladsgroup.json |
[production] |
| 12:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2122 (T323907)', diff saved to https://phabricator.wikimedia.org/P41707 and previous config saved to /var/cache/conftool/dbconfig/20221129-123134-ladsgroup.json |
[production] |
| 12:31 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance |
[production] |
| 12:31 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance |
[production] |
| 12:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2121 (T323907)', diff saved to https://phabricator.wikimedia.org/P41706 and previous config saved to /var/cache/conftool/dbconfig/20221129-123113-ladsgroup.json |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1132 (T321126)', diff saved to https://phabricator.wikimedia.org/P41705 and previous config saved to /var/cache/conftool/dbconfig/20221129-122657-marostegui.json |
[production] |
| 12:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1132 (T321126)', diff saved to https://phabricator.wikimedia.org/P41704 and previous config saved to /var/cache/conftool/dbconfig/20221129-122436-marostegui.json |
[production] |
| 12:24 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1132.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1132.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321126)', diff saved to https://phabricator.wikimedia.org/P41703 and previous config saved to /var/cache/conftool/dbconfig/20221129-122414-marostegui.json |
[production] |
| 12:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P41702 and previous config saved to /var/cache/conftool/dbconfig/20221129-122108-ladsgroup.json |
[production] |
| 12:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P41701 and previous config saved to /var/cache/conftool/dbconfig/20221129-121606-ladsgroup.json |
[production] |
| 12:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T322618)', diff saved to https://phabricator.wikimedia.org/P41700 and previous config saved to /var/cache/conftool/dbconfig/20221129-121354-ladsgroup.json |
[production] |
| 12:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P41699 and previous config saved to /var/cache/conftool/dbconfig/20221129-120907-marostegui.json |
[production] |
| 12:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T323907)', diff saved to https://phabricator.wikimedia.org/P41698 and previous config saved to /var/cache/conftool/dbconfig/20221129-120601-ladsgroup.json |
[production] |
| 12:05 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: sync |
[production] |
| 12:04 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: sync |
[production] |
| 12:03 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: sync |
[production] |
| 12:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P41697 and previous config saved to /var/cache/conftool/dbconfig/20221129-120100-ladsgroup.json |
[production] |
| 11:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P41696 and previous config saved to /var/cache/conftool/dbconfig/20221129-115847-ladsgroup.json |
[production] |
| 11:54 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host grafana2001.codfw.wmnet |
[production] |
| 11:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1128', diff saved to https://phabricator.wikimedia.org/P41695 and previous config saved to /var/cache/conftool/dbconfig/20221129-115401-marostegui.json |
[production] |
| 11:53 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: sync |
[production] |
| 11:53 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetdb2003.codfw.wmnet |
[production] |
| 11:47 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host grafana2001.codfw.wmnet |
[production] |
| 11:47 |
<marostegui> |
Drop scholarships database from m2 T243037 |
[production] |
| 11:47 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host puppetdb2003.codfw.wmnet |
[production] |
| 11:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2121 (T323907)', diff saved to https://phabricator.wikimedia.org/P41694 and previous config saved to /var/cache/conftool/dbconfig/20221129-114553-ladsgroup.json |
[production] |
| 11:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P41693 and previous config saved to /var/cache/conftool/dbconfig/20221129-114341-ladsgroup.json |
[production] |
| 11:43 |
<godog> |
+100G to global/prometheus in eqiad |
[production] |
| 11:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1128 (T321126)', diff saved to https://phabricator.wikimedia.org/P41692 and previous config saved to /var/cache/conftool/dbconfig/20221129-113854-marostegui.json |
[production] |
| 11:37 |
<moritzm> |
uploaded ferm 2.5.1-1.1+wmf11u1 to apt.wikimedia.org/bookworm (rebasing our systemd startup fixes to what's in bookworm) T321783 |
[production] |
| 11:37 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: sync |
[production] |
| 11:37 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: sync |
[production] |