|
2022-04-11
ยง
|
| 13:04 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@f0a1656] (hadoop-test): Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] |
[production] |
| 13:03 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@f0a1656] (thin): Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] (duration: 00m 07s) |
[production] |
| 13:03 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@f0a1656] (thin): Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] |
[production] |
| 12:58 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@f0a1656]: Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] (duration: 20m 23s) |
[production] |
| 12:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P24418 and previous config saved to /var/cache/conftool/dbconfig/20220411-125411-ladsgroup.json |
[production] |
| 12:48 |
<aqu@deploy1002> |
Finished deploy [airflow-dags/analytics@cae0024]: T302876_migrate_mediarequest_to_airflow [airflow-dags/analytics@cae0024] (duration: 00m 32s) |
[production] |
| 12:47 |
<aqu@deploy1002> |
Started deploy [airflow-dags/analytics@cae0024]: T302876_migrate_mediarequest_to_airflow [airflow-dags/analytics@cae0024] |
[production] |
| 12:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24417 and previous config saved to /var/cache/conftool/dbconfig/20220411-123906-ladsgroup.json |
[production] |
| 12:37 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@f0a1656]: Migrate mediarequest hourly from Oozie to Airflow [analytics/refinery@f0a1656] |
[production] |
| 12:36 |
<aqu> |
About to deploy analytics/refinery "Migrate mediarequest hourly from Oozie to Airflow" |
[production] |
| 12:31 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1151.eqiad.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:31 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:30:00 on db1151.eqiad.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:26 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2142.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:25 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:30:00 on db2142.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 12:25 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Rebooting x2 codfw primary T303174 |
[production] |
| 12:25 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Rebooting x2 codfw primary T303174 |
[production] |
| 12:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T297189)', diff saved to https://phabricator.wikimedia.org/P24416 and previous config saved to /var/cache/conftool/dbconfig/20220411-122220-marostegui.json |
[production] |
| 12:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:22 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T297189)', diff saved to https://phabricator.wikimedia.org/P24415 and previous config saved to /var/cache/conftool/dbconfig/20220411-122212-marostegui.json |
[production] |
| 12:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P24414 and previous config saved to /var/cache/conftool/dbconfig/20220411-120707-marostegui.json |
[production] |
| 12:02 |
<kevinbazira@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . |
[production] |
| 11:56 |
<kevinbazira@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . |
[production] |
| 11:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P24413 and previous config saved to /var/cache/conftool/dbconfig/20220411-115202-marostegui.json |
[production] |
| 11:46 |
<topranks> |
Adjust loopback filter on asw1-b12-drmrs to align with CR router config. T304553. |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24412 and previous config saved to /var/cache/conftool/dbconfig/20220411-114053-ladsgroup.json |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24411 and previous config saved to /var/cache/conftool/dbconfig/20220411-114041-ladsgroup.json |
[production] |
| 11:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T297189)', diff saved to https://phabricator.wikimedia.org/P24410 and previous config saved to /var/cache/conftool/dbconfig/20220411-113657-marostegui.json |
[production] |
| 11:34 |
<topranks> |
Adjust loopback filter on cr3-ulsfo to align with L3 switch config. T304553. |
[production] |
| 11:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1119', diff saved to https://phabricator.wikimedia.org/P24409 and previous config saved to /var/cache/conftool/dbconfig/20220411-112825-root.json |
[production] |
| 11:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1106', diff saved to https://phabricator.wikimedia.org/P24408 and previous config saved to /var/cache/conftool/dbconfig/20220411-112741-root.json |
[production] |
| 11:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P24407 and previous config saved to /var/cache/conftool/dbconfig/20220411-112536-ladsgroup.json |
[production] |
| 11:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1106', diff saved to https://phabricator.wikimedia.org/P24406 and previous config saved to /var/cache/conftool/dbconfig/20220411-112452-root.json |
[production] |
| 11:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1106', diff saved to https://phabricator.wikimedia.org/P24405 and previous config saved to /var/cache/conftool/dbconfig/20220411-112229-root.json |
[production] |
| 11:18 |
<btullis@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster |
[production] |
| 11:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P24404 and previous config saved to /var/cache/conftool/dbconfig/20220411-111030-ladsgroup.json |
[production] |
| 10:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298565)', diff saved to https://phabricator.wikimedia.org/P24403 and previous config saved to /var/cache/conftool/dbconfig/20220411-105525-ladsgroup.json |
[production] |
| 10:41 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2121.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 10:41 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:30:00 on db2121.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 10:38 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2121.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 10:38 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:30:00 on db2121.codfw.wmnet with reason: Rebooting for T303174 |
[production] |
| 10:37 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 11 hosts with reason: Rebooting primary T303174 |
[production] |
| 10:37 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 11 hosts with reason: Rebooting primary T303174 |
[production] |
| 10:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1158 (T297189)', diff saved to https://phabricator.wikimedia.org/P24402 and previous config saved to /var/cache/conftool/dbconfig/20220411-103336-marostegui.json |
[production] |
| 10:33 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:33 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:33 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:33 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |