|
2023-02-09
§
|
| 14:25 |
<dcausse@deploy1002> |
Started deploy [wikimedia/discovery/analytics@dc3cd56]: T329089: proper reconciliation of missed page-undelete events |
[production] |
| 14:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P44029 and previous config saved to /var/cache/conftool/dbconfig/20230209-142321-ladsgroup.json |
[production] |
| 14:22 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2430.codfw.wmnet with reason: host reimage |
[production] |
| 14:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P44028 and previous config saved to /var/cache/conftool/dbconfig/20230209-142157-marostegui.json |
[production] |
| 14:21 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2432.codfw.wmnet with OS buster |
[production] |
| 14:19 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2430.codfw.wmnet with reason: host reimage |
[production] |
| 14:17 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2429.codfw.wmnet with reason: host reimage |
[production] |
| 14:14 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2431.codfw.wmnet with OS buster |
[production] |
| 14:14 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2429.codfw.wmnet with reason: host reimage |
[production] |
| 14:14 |
<dcausse> |
T329089: re-playing detected inconsistencies (missing mediawiki.page-undelete events) from 2022-10-31 to 2023-02-07 to WDQS |
[production] |
| 14:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P44027 and previous config saved to /var/cache/conftool/dbconfig/20230209-141247-marostegui.json |
[production] |
| 14:09 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@mwdebug1001:~$ mwscript namespaceDupes.php shnwikibooks --fix | tee T328634-1-unpatched.out # T328634 – finished successfully, to my surprise |
[production] |
| 14:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P44026 and previous config saved to /var/cache/conftool/dbconfig/20230209-140815-ladsgroup.json |
[production] |
| 14:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2158 (T329203)', diff saved to https://phabricator.wikimedia.org/P44025 and previous config saved to /var/cache/conftool/dbconfig/20230209-140650-marostegui.json |
[production] |
| 14:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2158 (T329203)', diff saved to https://phabricator.wikimedia.org/P44024 and previous config saved to /var/cache/conftool/dbconfig/20230209-140124-marostegui.json |
[production] |
| 14:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance |
[production] |
| 14:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance |
[production] |
| 14:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance |
[production] |
| 14:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance |
[production] |
| 14:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2151 (T329203)', diff saved to https://phabricator.wikimedia.org/P44023 and previous config saved to /var/cache/conftool/dbconfig/20230209-140059-marostegui.json |
[production] |
| 13:58 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2430.codfw.wmnet with OS buster |
[production] |
| 13:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P44022 and previous config saved to /var/cache/conftool/dbconfig/20230209-135741-marostegui.json |
[production] |
| 13:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1113:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P44021 and previous config saved to /var/cache/conftool/dbconfig/20230209-135441-marostegui.json |
[production] |
| 13:54 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2429.codfw.wmnet with OS buster |
[production] |
| 13:54 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:54 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110 (T328817)', diff saved to https://phabricator.wikimedia.org/P44020 and previous config saved to /var/cache/conftool/dbconfig/20230209-135420-marostegui.json |
[production] |
| 13:53 |
<joal@deploy1002> |
Finished deploy [airflow-dags/analytics@fbebd61]: Update analytics actor dags spark resources (duration: 00m 13s) |
[production] |
| 13:53 |
<joal@deploy1002> |
Started deploy [airflow-dags/analytics@fbebd61]: Update analytics actor dags spark resources |
[production] |
| 13:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T328255)', diff saved to https://phabricator.wikimedia.org/P44019 and previous config saved to /var/cache/conftool/dbconfig/20230209-135309-ladsgroup.json |
[production] |
| 13:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P44018 and previous config saved to /var/cache/conftool/dbconfig/20230209-134553-marostegui.json |
[production] |
| 13:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2109 (T328255)', diff saved to https://phabricator.wikimedia.org/P44017 and previous config saved to /var/cache/conftool/dbconfig/20230209-134343-ladsgroup.json |
[production] |
| 13:43 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 13:43 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 13:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105 (T328255)', diff saved to https://phabricator.wikimedia.org/P44016 and previous config saved to /var/cache/conftool/dbconfig/20230209-134322-ladsgroup.json |
[production] |
| 13:40 |
<elukey> |
restart prometheus-statsd-exporter on ores nodes to pick up label change - T325763 |
[production] |
| 13:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P44014 and previous config saved to /var/cache/conftool/dbconfig/20230209-133914-marostegui.json |
[production] |
| 13:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P44013 and previous config saved to /var/cache/conftool/dbconfig/20230209-133046-marostegui.json |
[production] |
| 13:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P44012 and previous config saved to /var/cache/conftool/dbconfig/20230209-132815-ladsgroup.json |
[production] |
| 13:27 |
<hashar> |
phab2002: manually stopped `phd` service. It can't start due to the MariaDB server being set read-only and failed to start every 10 seconds since forever |
[production] |
| 13:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P44011 and previous config saved to /var/cache/conftool/dbconfig/20230209-132407-marostegui.json |
[production] |
| 13:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2151 (T329203)', diff saved to https://phabricator.wikimedia.org/P44010 and previous config saved to /var/cache/conftool/dbconfig/20230209-131540-marostegui.json |
[production] |
| 13:14 |
<moritzm> |
restarting Exim on MXes to pick up OpenSSL update |
[production] |
| 13:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P44009 and previous config saved to /var/cache/conftool/dbconfig/20230209-131309-ladsgroup.json |
[production] |
| 13:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2151 (T329203)', diff saved to https://phabricator.wikimedia.org/P44008 and previous config saved to /var/cache/conftool/dbconfig/20230209-131010-marostegui.json |
[production] |
| 13:10 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance |
[production] |
| 13:09 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance |
[production] |
| 13:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110 (T328817)', diff saved to https://phabricator.wikimedia.org/P44007 and previous config saved to /var/cache/conftool/dbconfig/20230209-130901-marostegui.json |
[production] |
| 13:06 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance |
[production] |
| 13:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance |
[production] |