|
2022-11-24
ยง
|
| 09:40 |
<filippo@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: graphite2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1001" |
[production] |
| 09:38 |
<filippo@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:33 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts graphite2003.codfw.wmnet |
[production] |
| 09:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1111 (T321126)', diff saved to https://phabricator.wikimedia.org/P40962 and previous config saved to /var/cache/conftool/dbconfig/20221124-092912-marostegui.json |
[production] |
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1111 (T321126)', diff saved to https://phabricator.wikimedia.org/P40961 and previous config saved to /var/cache/conftool/dbconfig/20221124-092804-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1111.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1111.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1104 (T321126)', diff saved to https://phabricator.wikimedia.org/P40960 and previous config saved to /var/cache/conftool/dbconfig/20221124-092742-marostegui.json |
[production] |
| 09:26 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 09:26 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 09:24 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 09:23 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 09:22 |
<oblivian@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 09:20 |
<oblivian@deploy1002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P40959 and previous config saved to /var/cache/conftool/dbconfig/20221124-091236-marostegui.json |
[production] |
| 09:10 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T323214)', diff saved to https://phabricator.wikimedia.org/P40958 and previous config saved to /var/cache/conftool/dbconfig/20221124-091017-ladsgroup.json |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1104', diff saved to https://phabricator.wikimedia.org/P40957 and previous config saved to /var/cache/conftool/dbconfig/20221124-085729-marostegui.json |
[production] |
| 08:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P40956 and previous config saved to /var/cache/conftool/dbconfig/20221124-085511-ladsgroup.json |
[production] |
| 08:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1104 (T321126)', diff saved to https://phabricator.wikimedia.org/P40955 and previous config saved to /var/cache/conftool/dbconfig/20221124-084223-marostegui.json |
[production] |
| 08:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1104 (T321126)', diff saved to https://phabricator.wikimedia.org/P40954 and previous config saved to /var/cache/conftool/dbconfig/20221124-084015-marostegui.json |
[production] |
| 08:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1104.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P40953 and previous config saved to /var/cache/conftool/dbconfig/20221124-084004-ladsgroup.json |
[production] |
| 08:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1104.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40952 and previous config saved to /var/cache/conftool/dbconfig/20221124-083954-marostegui.json |
[production] |
| 08:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T323214)', diff saved to https://phabricator.wikimedia.org/P40951 and previous config saved to /var/cache/conftool/dbconfig/20221124-082458-ladsgroup.json |
[production] |
| 08:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P40950 and previous config saved to /var/cache/conftool/dbconfig/20221124-082447-marostegui.json |
[production] |
| 08:13 |
<moritzm> |
installing tomcat9 security updates |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P40949 and previous config saved to /var/cache/conftool/dbconfig/20221124-080941-marostegui.json |
[production] |
| 08:04 |
<moritzm> |
rebalance Ganeti group A/codfw following reboots |
[production] |
| 07:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40948 and previous config saved to /var/cache/conftool/dbconfig/20221124-075434-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1101:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40947 and previous config saved to /var/cache/conftool/dbconfig/20221124-075226-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T321126)', diff saved to https://phabricator.wikimedia.org/P40946 and previous config saved to /var/cache/conftool/dbconfig/20221124-075205-marostegui.json |
[production] |
| 07:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T323214)', diff saved to https://phabricator.wikimedia.org/P40945 and previous config saved to /var/cache/conftool/dbconfig/20221124-074517-ladsgroup.json |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P40944 and previous config saved to /var/cache/conftool/dbconfig/20221124-073658-marostegui.json |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1201 (T323214)', diff saved to https://phabricator.wikimedia.org/P40943 and previous config saved to /var/cache/conftool/dbconfig/20221124-073637-ladsgroup.json |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T323214)', diff saved to https://phabricator.wikimedia.org/P40942 and previous config saved to /var/cache/conftool/dbconfig/20221124-073616-ladsgroup.json |
[production] |
| 07:30 |
<phedenskog@deploy1002> |
Finished deploy [performance/navtiming@e421904]: (no justification provided) (duration: 00m 08s) |
[production] |
| 07:30 |
<phedenskog@deploy1002> |
Started deploy [performance/navtiming@e421904]: (no justification provided) |
[production] |
| 07:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P40941 and previous config saved to /var/cache/conftool/dbconfig/20221124-073011-ladsgroup.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P40940 and previous config saved to /var/cache/conftool/dbconfig/20221124-072152-marostegui.json |
[production] |
| 07:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P40939 and previous config saved to /var/cache/conftool/dbconfig/20221124-072110-ladsgroup.json |
[production] |
| 07:15 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P40938 and previous config saved to /var/cache/conftool/dbconfig/20221124-071504-ladsgroup.json |
[production] |