|
2024-01-29
ยง
|
| 12:06 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1231.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:00 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-role for role: analytics_cluster::airflow::wmde |
[production] |
| 12:00 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:59 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1224 (T355609)', diff saved to https://phabricator.wikimedia.org/P55790 and previous config saved to /var/cache/conftool/dbconfig/20240129-115953-marostegui.json |
[production] |
| 11:53 |
<Dreamy_Jazz> |
Running mwscript maintenance/sql.php --wiki=testwiki --wikidb=centralauth ~/T354700-create-table-global.sql for T354700 |
[production] |
| 11:45 |
<Dreamy_Jazz> |
sql.php finished for T354700 |
[production] |
| 11:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P55789 and previous config saved to /var/cache/conftool/dbconfig/20240129-114446-marostegui.json |
[production] |
| 11:41 |
<Dreamy_Jazz> |
T354700 - Running `foreachwiki maintenance/sql.php ~/T354700-create-table.sql` |
[production] |
| 11:39 |
<Dreamy_Jazz> |
T354700 - Ran mwscript maintenance/sql.php --wiki=testwiki ~/T354700-create-table.sql |
[production] |
| 11:38 |
<moritzm> |
upload ganeti 3.0.2-3+wmf1 (bookworm package of Ganeti plus backport for SSL chain handling in RAPI) to apt.wikimedia.org T300152 |
[production] |
| 11:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P55788 and previous config saved to /var/cache/conftool/dbconfig/20240129-112940-marostegui.json |
[production] |
| 11:28 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-airflow1007.eqiad.wmnet with OS bullseye |
[production] |
| 11:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1224 (T355609)', diff saved to https://phabricator.wikimedia.org/P55787 and previous config saved to /var/cache/conftool/dbconfig/20240129-111434-marostegui.json |
[production] |
| 11:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1224 (T355609)', diff saved to https://phabricator.wikimedia.org/P55786 and previous config saved to /var/cache/conftool/dbconfig/20240129-110955-marostegui.json |
[production] |
| 11:09 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1224.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:09 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1224.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1213:3316 (T355609)', diff saved to https://phabricator.wikimedia.org/P55785 and previous config saved to /var/cache/conftool/dbconfig/20240129-110933-marostegui.json |
[production] |
| 11:05 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-airflow1007.eqiad.wmnet with reason: host reimage |
[production] |
| 11:01 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-airflow1007.eqiad.wmnet with reason: host reimage |
[production] |
| 10:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1213:3316', diff saved to https://phabricator.wikimedia.org/P55784 and previous config saved to /var/cache/conftool/dbconfig/20240129-105427-marostegui.json |
[production] |
| 10:53 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1054.eqiad.wmnet |
[production] |
| 10:53 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2054.codfw.wmnet |
[production] |
| 10:47 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host mc2054.codfw.wmnet |
[production] |
| 10:47 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host mc1054.eqiad.wmnet |
[production] |
| 10:47 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reimage for host an-airflow1007.eqiad.wmnet with OS bullseye |
[production] |
| 10:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1213:3316', diff saved to https://phabricator.wikimedia.org/P55783 and previous config saved to /var/cache/conftool/dbconfig/20240129-103920-marostegui.json |
[production] |
| 10:38 |
<arnaudb@cumin1002> |
END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) Will create a clone of db2169.codfw.wmnet onto db2194.codfw.wmnet |
[production] |
| 10:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.clone Will create a clone of db2169.codfw.wmnet onto db2194.codfw.wmnet |
[production] |
| 10:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1213:3316 (T355609)', diff saved to https://phabricator.wikimedia.org/P55782 and previous config saved to /var/cache/conftool/dbconfig/20240129-102414-marostegui.json |
[production] |
| 10:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1213:3316 (T355609)', diff saved to https://phabricator.wikimedia.org/P55781 and previous config saved to /var/cache/conftool/dbconfig/20240129-101757-marostegui.json |
[production] |
| 10:17 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T355609)', diff saved to https://phabricator.wikimedia.org/P55780 and previous config saved to /var/cache/conftool/dbconfig/20240129-101735-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P55779 and previous config saved to /var/cache/conftool/dbconfig/20240129-100229-marostegui.json |
[production] |
| 10:00 |
<moritzm> |
upload prometheus-ganeti-exporter 0.3+deb12u1 to apt.wikimedia.org T300152 |
[production] |
| 09:56 |
<XioNoX> |
enable Puppet on all the ganeti servers for CR990968 deployment - T300152 |
[production] |
| 09:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P55778 and previous config saved to /var/cache/conftool/dbconfig/20240129-094722-marostegui.json |
[production] |
| 09:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T355609)', diff saved to https://phabricator.wikimedia.org/P55777 and previous config saved to /var/cache/conftool/dbconfig/20240129-093216-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1201 (T355609)', diff saved to https://phabricator.wikimedia.org/P55776 and previous config saved to /var/cache/conftool/dbconfig/20240129-092724-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T355609)', diff saved to https://phabricator.wikimedia.org/P55775 and previous config saved to /var/cache/conftool/dbconfig/20240129-092702-marostegui.json |
[production] |
| 09:17 |
<godog> |
mark for deletetion and cleanup replicated thanos blocks for prometheus=ops, older than 3 months, all resolutions - T351927 |
[production] |
| 09:13 |
<moritzm> |
upgrading python-pymysql in S7 DB hosts to 1.0.2-2~wmf11u1 T355531 |
[production] |
| 09:13 |
<XioNoX> |
disable Puppet on all the ganeti servers for CR990968 deployment - T300152 |
[production] |
| 09:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P55773 and previous config saved to /var/cache/conftool/dbconfig/20240129-091156-marostegui.json |
[production] |
| 08:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P55772 and previous config saved to /var/cache/conftool/dbconfig/20240129-085649-marostegui.json |
[production] |
| 08:46 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:993489|Revert "ProductionServices.php: Promote pc2014"]] (duration: 17m 13s) |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T355609)', diff saved to https://phabricator.wikimedia.org/P55771 and previous config saved to /var/cache/conftool/dbconfig/20240129-084143-marostegui.json |
[production] |