|
2024-02-02
ยง
|
| 10:33 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-role for role: analytics_cluster::airflow::platform_eng |
[production] |
| 10:32 |
<Emperor> |
restart codfw swift-container (-b 1 -s 3) |
[production] |
| 10:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2155 (T355609)', diff saved to https://phabricator.wikimedia.org/P56108 and previous config saved to /var/cache/conftool/dbconfig/20240202-102943-marostegui.json |
[production] |
| 10:29 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 10:29 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 10:29 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 10:29 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 10:28 |
<Emperor> |
restart codfw swift-account (-b 1 -s 3) |
[production] |
| 10:27 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reimage for host an-airflow1002.eqiad.wmnet with OS bullseye |
[production] |
| 10:20 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-airflow1004.eqiad.wmnet with OS bullseye |
[production] |
| 10:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T355609)', diff saved to https://phabricator.wikimedia.org/P56107 and previous config saved to /var/cache/conftool/dbconfig/20240202-101311-marostegui.json |
[production] |
| 10:01 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-airflow1004.eqiad.wmnet with reason: host reimage |
[production] |
| 09:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P56106 and previous config saved to /var/cache/conftool/dbconfig/20240202-095805-marostegui.json |
[production] |
| 09:56 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-airflow1004.eqiad.wmnet with reason: host reimage |
[production] |
| 09:45 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reimage for host an-airflow1004.eqiad.wmnet with OS bullseye |
[production] |
| 09:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3314', diff saved to https://phabricator.wikimedia.org/P56104 and previous config saved to /var/cache/conftool/dbconfig/20240202-094258-marostegui.json |
[production] |
| 09:36 |
<akosiaris@deploy2002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:36 |
<akosiaris@deploy2002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 09:35 |
<akosiaris@deploy2002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:35 |
<akosiaris@deploy2002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 09:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137:3314 (T355609)', diff saved to https://phabricator.wikimedia.org/P56103 and previous config saved to /var/cache/conftool/dbconfig/20240202-092752-marostegui.json |
[production] |
| 09:18 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe |
[production] |
| 09:09 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe |
[production] |
| 09:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2137:3314 (T355609)', diff saved to https://phabricator.wikimedia.org/P56102 and previous config saved to /var/cache/conftool/dbconfig/20240202-090209-marostegui.json |
[production] |
| 09:02 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 09:01 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 09:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2155 (T355609)', diff saved to https://phabricator.wikimedia.org/P56101 and previous config saved to /var/cache/conftool/dbconfig/20240202-090146-marostegui.json |
[production] |
| 08:57 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on P{ms-fe1011.eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 08:57 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on P{ms-fe1011.eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 08:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P56100 and previous config saved to /var/cache/conftool/dbconfig/20240202-084640-marostegui.json |
[production] |
| 08:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P56099 and previous config saved to /var/cache/conftool/dbconfig/20240202-083133-marostegui.json |
[production] |
| 08:30 |
<tstarling@deploy2002> |
Synchronized wmf-config/CommonSettings.php: Enable UrlShortener QR code everywhere (T348487) (duration: 07m 23s) |
[production] |
| 08:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2155 (T355609)', diff saved to https://phabricator.wikimedia.org/P56098 and previous config saved to /var/cache/conftool/dbconfig/20240202-081626-marostegui.json |
[production] |
| 07:18 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org |
[production] |
| 07:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2155 (T355609)', diff saved to https://phabricator.wikimedia.org/P56097 and previous config saved to /var/cache/conftool/dbconfig/20240202-071555-marostegui.json |
[production] |
| 07:15 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 07:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T355609)', diff saved to https://phabricator.wikimedia.org/P56096 and previous config saved to /var/cache/conftool/dbconfig/20240202-071516-marostegui.json |
[production] |
| 07:12 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org |
[production] |
| 07:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P56095 and previous config saved to /var/cache/conftool/dbconfig/20240202-070009-marostegui.json |
[production] |
| 06:58 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org |
[production] |
| 06:53 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org |
[production] |
| 06:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P56094 and previous config saved to /var/cache/conftool/dbconfig/20240202-064502-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Change db1163 weight', diff saved to https://phabricator.wikimedia.org/P56093 and previous config saved to /var/cache/conftool/dbconfig/20240202-063858-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Change db1163 weight', diff saved to https://phabricator.wikimedia.org/P56092 and previous config saved to /var/cache/conftool/dbconfig/20240202-063844-marostegui.json |
[production] |
| 06:14 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1106.eqiad.wmnet |
[production] |
| 06:14 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 06:14 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1106.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002" |
[production] |