|
2023-03-09
§
|
| 02:59 |
<sukhe> |
run keyholder arm on acmechief2001 |
[production] |
| 02:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P45608 and previous config saved to /var/cache/conftool/dbconfig/20230309-024917-marostegui.json |
[production] |
| 02:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P45607 and previous config saved to /var/cache/conftool/dbconfig/20230309-023411-marostegui.json |
[production] |
| 02:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3318 (T329260)', diff saved to https://phabricator.wikimedia.org/P45606 and previous config saved to /var/cache/conftool/dbconfig/20230309-021905-marostegui.json |
[production] |
| 01:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2168:3318 (T329260)', diff saved to https://phabricator.wikimedia.org/P45604 and previous config saved to /var/cache/conftool/dbconfig/20230309-015831-marostegui.json |
[production] |
| 01:58 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance |
[production] |
| 01:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance |
[production] |
| 01:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T329260)', diff saved to https://phabricator.wikimedia.org/P45603 and previous config saved to /var/cache/conftool/dbconfig/20230309-015810-marostegui.json |
[production] |
| 01:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P45602 and previous config saved to /var/cache/conftool/dbconfig/20230309-014303-marostegui.json |
[production] |
| 01:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P45601 and previous config saved to /var/cache/conftool/dbconfig/20230309-012757-marostegui.json |
[production] |
| 01:18 |
<ebernhardson@deploy2002> |
Finished deploy [airflow-dags/search@558da74]: correct eventgate datacenter partitioning in sensors (duration: 00m 13s) |
[production] |
| 01:18 |
<ebernhardson@deploy2002> |
Started deploy [airflow-dags/search@558da74]: correct eventgate datacenter partitioning in sensors |
[production] |
| 01:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T329260)', diff saved to https://phabricator.wikimedia.org/P45600 and previous config saved to /var/cache/conftool/dbconfig/20230309-011251-marostegui.json |
[production] |
| 00:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2167:3318 (T329260)', diff saved to https://phabricator.wikimedia.org/P45599 and previous config saved to /var/cache/conftool/dbconfig/20230309-005220-marostegui.json |
[production] |
| 00:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance |
[production] |
| 00:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance |
[production] |
| 00:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2166 (T329260)', diff saved to https://phabricator.wikimedia.org/P45598 and previous config saved to /var/cache/conftool/dbconfig/20230309-005210-marostegui.json |
[production] |
| 00:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P45597 and previous config saved to /var/cache/conftool/dbconfig/20230309-003703-marostegui.json |
[production] |
| 00:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P45596 and previous config saved to /var/cache/conftool/dbconfig/20230309-002157-marostegui.json |
[production] |
| 00:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2166 (T329260)', diff saved to https://phabricator.wikimedia.org/P45594 and previous config saved to /var/cache/conftool/dbconfig/20230309-000651-marostegui.json |
[production] |
|
2023-03-08
§
|
| 23:49 |
<zabe@deploy2002> |
Finished scap: T308932 (duration: 07m 15s) |
[production] |
| 23:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2166 (T329260)', diff saved to https://phabricator.wikimedia.org/P45593 and previous config saved to /var/cache/conftool/dbconfig/20230308-234534-marostegui.json |
[production] |
| 23:45 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance |
[production] |
| 23:45 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance |
[production] |
| 23:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T329260)', diff saved to https://phabricator.wikimedia.org/P45592 and previous config saved to /var/cache/conftool/dbconfig/20230308-234502-marostegui.json |
[production] |
| 23:42 |
<zabe@deploy2002> |
Started scap: T308932 |
[production] |
| 23:42 |
<ebernhardson@deploy2002> |
Finished deploy [airflow-dags/search@29f73a4]: update virtualenv entry_points to use relative paths (duration: 00m 14s) |
[production] |
| 23:42 |
<ebernhardson@deploy2002> |
Started deploy [airflow-dags/search@29f73a4]: update virtualenv entry_points to use relative paths |
[production] |
| 23:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P45591 and previous config saved to /var/cache/conftool/dbconfig/20230308-232956-marostegui.json |
[production] |
| 23:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P45590 and previous config saved to /var/cache/conftool/dbconfig/20230308-231449-marostegui.json |
[production] |
| 22:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T329260)', diff saved to https://phabricator.wikimedia.org/P45589 and previous config saved to /var/cache/conftool/dbconfig/20230308-225943-marostegui.json |
[production] |
| 22:44 |
<hashar> |
Upgrading CI Jenkins |
[production] |
| 22:42 |
<tgr> |
UTC late deploys done |
[production] |
| 22:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2164 (T329260)', diff saved to https://phabricator.wikimedia.org/P45588 and previous config saved to /var/cache/conftool/dbconfig/20230308-224044-marostegui.json |
[production] |
| 22:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 22:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 22:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 22:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 22:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T329260)', diff saved to https://phabricator.wikimedia.org/P45587 and previous config saved to /var/cache/conftool/dbconfig/20230308-224018-marostegui.json |
[production] |
| 22:39 |
<tgr@deploy2002> |
Finished scap: Backport for [[gerrit:895776|Leveling up: check if the task type is registered before increasing its edit count (T331524)]], [[gerrit:895777|Leveling up: check if the task type is registered before increasing its edit count (T331524)]] (duration: 08m 31s) |
[production] |
| 22:32 |
<tgr@deploy2002> |
tgr: Backport for [[gerrit:895776|Leveling up: check if the task type is registered before increasing its edit count (T331524)]], [[gerrit:895777|Leveling up: check if the task type is registered before increasing its edit count (T331524)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 22:30 |
<tgr@deploy2002> |
Started scap: Backport for [[gerrit:895776|Leveling up: check if the task type is registered before increasing its edit count (T331524)]], [[gerrit:895777|Leveling up: check if the task type is registered before increasing its edit count (T331524)]] |
[production] |
| 22:29 |
<tgr@deploy2002> |
Finished scap: Backport for [[gerrit:895778|maintenance: Adjust query builder to account for no secondary namespaces (T321983 T331412)]], [[gerrit:895779|maintenance: Adjust query builder to account for no secondary namespaces (T321983 T331412)]] (duration: 07m 43s) |
[production] |
| 22:27 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1001.eqiad.wmnet |
[production] |
| 22:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P45586 and previous config saved to /var/cache/conftool/dbconfig/20230308-222512-marostegui.json |
[production] |
| 22:23 |
<tgr@deploy2002> |
tgr: Backport for [[gerrit:895778|maintenance: Adjust query builder to account for no secondary namespaces (T321983 T331412)]], [[gerrit:895779|maintenance: Adjust query builder to account for no secondary namespaces (T321983 T331412)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 22:21 |
<tgr@deploy2002> |
Started scap: Backport for [[gerrit:895778|maintenance: Adjust query builder to account for no secondary namespaces (T321983 T331412)]], [[gerrit:895779|maintenance: Adjust query builder to account for no secondary namespaces (T321983 T331412)]] |
[production] |
| 22:21 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet |
[production] |
| 22:20 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1001.eqiad.wmnet |
[production] |
| 22:12 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet |
[production] |