|
2022-04-11
ยง
|
| 13:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P24422 and previous config saved to /var/cache/conftool/dbconfig/20220411-135343-root.json |
[production] |
| 13:53 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.reimage for host ms-fe1012.eqiad.wmnet with OS bullseye |
[production] |
| 13:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T297189)', diff saved to https://phabricator.wikimedia.org/P24421 and previous config saved to /var/cache/conftool/dbconfig/20220411-134848-marostegui.json |
[production] |
| 13:24 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 14 hosts with reason: Maintenance |
[production] |
| 13:24 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on 14 hosts with reason: Maintenance |
[production] |
| 13:24 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 13:24 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 13:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24420 and previous config saved to /var/cache/conftool/dbconfig/20220411-132422-ladsgroup.json |
[production] |
| 13:11 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@f0a1656] (hadoop-test): Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] (duration: 07m 00s) |
[production] |
| 13:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P24419 and previous config saved to /var/cache/conftool/dbconfig/20220411-130916-ladsgroup.json |
[production] |
| 13:04 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@f0a1656] (hadoop-test): Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] |
[production] |
| 13:03 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@f0a1656] (thin): Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] (duration: 00m 07s) |
[production] |
| 13:03 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@f0a1656] (thin): Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] |
[production] |
| 12:58 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@f0a1656]: Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] (duration: 20m 23s) |
[production] |
| 12:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P24418 and previous config saved to /var/cache/conftool/dbconfig/20220411-125411-ladsgroup.json |
[production] |
| 12:48 |
<aqu@deploy1002> |
Finished deploy [airflow-dags/analytics@cae0024]: T302876_migrate_mediarequest_to_airflow [airflow-dags/analytics@cae0024] (duration: 00m 32s) |
[production] |
| 12:47 |
<aqu@deploy1002> |
Started deploy [airflow-dags/analytics@cae0024]: T302876_migrate_mediarequest_to_airflow [airflow-dags/analytics@cae0024] |
[production] |
| 12:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24417 and previous config saved to /var/cache/conftool/dbconfig/20220411-123906-ladsgroup.json |
[production] |
| 12:37 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@f0a1656]: Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] |
[production] |
| 12:36 |
<aqu> |
About to deploy analytics/refinery "Migrate mediarequest hourly from Oozie to Airflow" |
[production] |
| 12:31 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1151.eqiad.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:31 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:30:00 on db1151.eqiad.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:26 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2142.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:25 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:30:00 on db2142.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:25 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Rebooting x2 codfw primary T303174 |
[production] |
| 12:25 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Rebooting x2 codfw primary T303174 |
[production] |
| 12:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T297189)', diff saved to https://phabricator.wikimedia.org/P24416 and previous config saved to /var/cache/conftool/dbconfig/20220411-122220-marostegui.json |
[production] |
| 12:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:22 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T297189)', diff saved to https://phabricator.wikimedia.org/P24415 and previous config saved to /var/cache/conftool/dbconfig/20220411-122212-marostegui.json |
[production] |
| 12:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P24414 and previous config saved to /var/cache/conftool/dbconfig/20220411-120707-marostegui.json |
[production] |
| 12:02 |
<kevinbazira@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . |
[production] |
| 11:56 |
<kevinbazira@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . |
[production] |
| 11:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P24413 and previous config saved to /var/cache/conftool/dbconfig/20220411-115202-marostegui.json |
[production] |
| 11:46 |
<topranks> |
Adjust loopback filter on asw1-b12-drmrs to align with CR router config. T304553. |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24412 and previous config saved to /var/cache/conftool/dbconfig/20220411-114053-ladsgroup.json |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24411 and previous config saved to /var/cache/conftool/dbconfig/20220411-114041-ladsgroup.json |
[production] |
| 11:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T297189)', diff saved to https://phabricator.wikimedia.org/P24410 and previous config saved to /var/cache/conftool/dbconfig/20220411-113657-marostegui.json |
[production] |
| 11:34 |
<topranks> |
Adjust loopback filter on cr3-ulsfo to align with L3 switch config. T304553. |
[production] |
| 11:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1119', diff saved to https://phabricator.wikimedia.org/P24409 and previous config saved to /var/cache/conftool/dbconfig/20220411-112825-root.json |
[production] |
| 11:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1106', diff saved to https://phabricator.wikimedia.org/P24408 and previous config saved to /var/cache/conftool/dbconfig/20220411-112741-root.json |
[production] |
| 11:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P24407 and previous config saved to /var/cache/conftool/dbconfig/20220411-112536-ladsgroup.json |
[production] |
| 11:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1106', diff saved to https://phabricator.wikimedia.org/P24406 and previous config saved to /var/cache/conftool/dbconfig/20220411-112452-root.json |
[production] |
| 11:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1106', diff saved to https://phabricator.wikimedia.org/P24405 and previous config saved to /var/cache/conftool/dbconfig/20220411-112229-root.json |
[production] |
| 11:18 |
<btullis@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster |
[production] |
| 11:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P24404 and previous config saved to /var/cache/conftool/dbconfig/20220411-111030-ladsgroup.json |
[production] |
| 10:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24403 and previous config saved to /var/cache/conftool/dbconfig/20220411-105525-ladsgroup.json |
[production] |
| 10:41 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2121.codfw.wmnet with reason: Rebooting for T303174 |
[production] |