|
2024-01-29
ยง
|
| 10:47 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host mc2054.codfw.wmnet |
[production] |
| 10:47 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host mc1054.eqiad.wmnet |
[production] |
| 10:47 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reimage for host an-airflow1007.eqiad.wmnet with OS bullseye |
[production] |
| 10:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1213:3316', diff saved to https://phabricator.wikimedia.org/P55783 and previous config saved to /var/cache/conftool/dbconfig/20240129-103920-marostegui.json |
[production] |
| 10:38 |
<arnaudb@cumin1002> |
END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) Will create a clone of db2169.codfw.wmnet onto db2194.codfw.wmnet |
[production] |
| 10:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.clone Will create a clone of db2169.codfw.wmnet onto db2194.codfw.wmnet |
[production] |
| 10:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1213:3316 (T355609)', diff saved to https://phabricator.wikimedia.org/P55782 and previous config saved to /var/cache/conftool/dbconfig/20240129-102414-marostegui.json |
[production] |
| 10:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1213:3316 (T355609)', diff saved to https://phabricator.wikimedia.org/P55781 and previous config saved to /var/cache/conftool/dbconfig/20240129-101757-marostegui.json |
[production] |
| 10:17 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T355609)', diff saved to https://phabricator.wikimedia.org/P55780 and previous config saved to /var/cache/conftool/dbconfig/20240129-101735-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P55779 and previous config saved to /var/cache/conftool/dbconfig/20240129-100229-marostegui.json |
[production] |
| 10:00 |
<moritzm> |
upload prometheus-ganeti-exporter 0.3+deb12u1 to apt.wikimedia.org T300152 |
[production] |
| 09:56 |
<XioNoX> |
enable Puppet on all the ganeti servers for CR990968 deployment - T300152 |
[production] |
| 09:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P55778 and previous config saved to /var/cache/conftool/dbconfig/20240129-094722-marostegui.json |
[production] |
| 09:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T355609)', diff saved to https://phabricator.wikimedia.org/P55777 and previous config saved to /var/cache/conftool/dbconfig/20240129-093216-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1201 (T355609)', diff saved to https://phabricator.wikimedia.org/P55776 and previous config saved to /var/cache/conftool/dbconfig/20240129-092724-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T355609)', diff saved to https://phabricator.wikimedia.org/P55775 and previous config saved to /var/cache/conftool/dbconfig/20240129-092702-marostegui.json |
[production] |
| 09:17 |
<godog> |
mark for deletetion and cleanup replicated thanos blocks for prometheus=ops, older than 3 months, all resolutions - T351927 |
[production] |
| 09:13 |
<moritzm> |
upgrading python-pymysql in S7 DB hosts to 1.0.2-2~wmf11u1 T355531 |
[production] |
| 09:13 |
<XioNoX> |
disable Puppet on all the ganeti servers for CR990968 deployment - T300152 |
[production] |
| 09:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P55773 and previous config saved to /var/cache/conftool/dbconfig/20240129-091156-marostegui.json |
[production] |
| 08:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P55772 and previous config saved to /var/cache/conftool/dbconfig/20240129-085649-marostegui.json |
[production] |
| 08:46 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:993489|Revert "ProductionServices.php: Promote pc2014"]] (duration: 17m 13s) |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T355609)', diff saved to https://phabricator.wikimedia.org/P55771 and previous config saved to /var/cache/conftool/dbconfig/20240129-084143-marostegui.json |
[production] |
| 08:39 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 08:39 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:993489|Revert "ProductionServices.php: Promote pc2014"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1187 (T355609)', diff saved to https://phabricator.wikimedia.org/P55770 and previous config saved to /var/cache/conftool/dbconfig/20240129-083627-marostegui.json |
[production] |
| 08:36 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:36 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T355609)', diff saved to https://phabricator.wikimedia.org/P55769 and previous config saved to /var/cache/conftool/dbconfig/20240129-083603-marostegui.json |
[production] |
| 08:29 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:993489|Revert "ProductionServices.php: Promote pc2014"]] |
[production] |
| 08:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P55768 and previous config saved to /var/cache/conftool/dbconfig/20240129-082057-marostegui.json |
[production] |
| 08:05 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P55767 and previous config saved to /var/cache/conftool/dbconfig/20240129-080550-marostegui.json |
[production] |
| 07:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T355609)', diff saved to https://phabricator.wikimedia.org/P55766 and previous config saved to /var/cache/conftool/dbconfig/20240129-075044-marostegui.json |
[production] |
| 07:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1180 (T355609)', diff saved to https://phabricator.wikimedia.org/P55765 and previous config saved to /var/cache/conftool/dbconfig/20240129-074541-marostegui.json |
[production] |
| 07:45 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:45 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T355609)', diff saved to https://phabricator.wikimedia.org/P55764 and previous config saved to /var/cache/conftool/dbconfig/20240129-074519-marostegui.json |
[production] |
| 07:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P55763 and previous config saved to /var/cache/conftool/dbconfig/20240129-073857-root.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P55762 and previous config saved to /var/cache/conftool/dbconfig/20240129-073012-marostegui.json |
[production] |
| 07:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P55761 and previous config saved to /var/cache/conftool/dbconfig/20240129-072352-root.json |
[production] |
| 07:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P55760 and previous config saved to /var/cache/conftool/dbconfig/20240129-071506-marostegui.json |
[production] |
| 07:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P55758 and previous config saved to /var/cache/conftool/dbconfig/20240129-070847-root.json |
[production] |
| 07:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T355609)', diff saved to https://phabricator.wikimedia.org/P55757 and previous config saved to /var/cache/conftool/dbconfig/20240129-065959-marostegui.json |
[production] |
| 06:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1168 (T355609)', diff saved to https://phabricator.wikimedia.org/P55756 and previous config saved to /var/cache/conftool/dbconfig/20240129-065450-marostegui.json |
[production] |
| 06:54 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:54 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |