|
2024-02-21
ยง
|
| 16:06 |
<jayme> |
imported prometheus-rsyslog-exporter 1.0.0+git20221110-1 to buster,bullseye,bookworm - T357616 |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2108 (T357189)', diff saved to https://phabricator.wikimedia.org/P57584 and previous config saved to /var/cache/conftool/dbconfig/20240221-160511-arnaudb.json |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 16:04 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 16:04 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/api-gateway: apply |
[production] |
| 16:04 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57583 and previous config saved to /var/cache/conftool/dbconfig/20240221-160423-root.json |
[production] |
| 16:03 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 16:03 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/api-gateway: apply |
[production] |
| 16:02 |
<topranks> |
Commencing network maintenance migrating servers to new switch codfw rack A8 T355874 |
[production] |
| 15:59 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: Migrating servers in codfw rack A7 to lsw1-a7-codfw |
[production] |
| 15:58 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: Migrating servers in codfw rack A7 to lsw1-a7-codfw |
[production] |
| 15:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P57582 and previous config saved to /var/cache/conftool/dbconfig/20240221-155839-marostegui.json |
[production] |
| 15:58 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a8-codfw.mgmt with reason: prepping for server uplink migration codfw rack a8 |
[production] |
| 15:57 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a8-codfw.mgmt with reason: prepping for server uplink migration codfw rack a8 |
[production] |
| 15:55 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 15:55 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 15:55 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 15:54 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/api-gateway: apply |
[production] |
| 15:52 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 15:51 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/api-gateway: apply |
[production] |
| 15:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57581 and previous config saved to /var/cache/conftool/dbconfig/20240221-154918-root.json |
[production] |
| 15:47 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2137.codfw.wmnet with reason: host reimage |
[production] |
| 15:46 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance |
[production] |
| 15:46 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance |
[production] |
| 15:44 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2137.codfw.wmnet with reason: host reimage |
[production] |
| 15:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P57580 and previous config saved to /var/cache/conftool/dbconfig/20240221-154333-marostegui.json |
[production] |
| 15:42 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:25:00 on db2106.codfw.wmnet with reason: T355874 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:25:00 on db2106.codfw.wmnet with reason: T355874 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:25:00 on db2146.codfw.wmnet with reason: T355874 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:25:00 on db2146.codfw.wmnet with reason: T355874 - Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw |
[production] |
| 15:40 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'T355874 - depooling db2146 db2106', diff saved to https://phabricator.wikimedia.org/P57579 and previous config saved to /var/cache/conftool/dbconfig/20240221-154056-arnaudb.json |
[production] |
| 15:39 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:39 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:39 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236 (T357189)', diff saved to https://phabricator.wikimedia.org/P57578 and previous config saved to /var/cache/conftool/dbconfig/20240221-153926-arnaudb.json |
[production] |
| 15:34 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57577 and previous config saved to /var/cache/conftool/dbconfig/20240221-153414-root.json |
[production] |
| 15:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T355609)', diff saved to https://phabricator.wikimedia.org/P57576 and previous config saved to /var/cache/conftool/dbconfig/20240221-152826-marostegui.json |
[production] |
| 15:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P57575 and previous config saved to /var/cache/conftool/dbconfig/20240221-152420-arnaudb.json |
[production] |
| 15:21 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.reimage for host db2137.codfw.wmnet with OS bookworm |
[production] |
| 15:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2026 (re)pooling @ 5%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57574 and previous config saved to /var/cache/conftool/dbconfig/20240221-151909-root.json |
[production] |
| 15:12 |
<hnowlan@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/thumbor: apply |
[production] |
| 15:12 |
<hnowlan@deploy2002> |
helmfile [codfw] START helmfile.d/services/thumbor: apply |
[production] |
| 15:10 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:10 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for private1-b-codfw - cmooney@cumin1002" |
[production] |
| 14:55 |
<TheresNoTime> |
UTC afternoon backport window done |
[production] |
| 14:54 |
<hnowlan@deploy2002> |
helmfile [codfw] START helmfile.d/services/thumbor: apply |
[production] |
| 14:54 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 14:54 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for private1-a-codfw - cmooney@cumin1002" |
[production] |
| 14:54 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1236 (T357189)', diff saved to https://phabricator.wikimedia.org/P57570 and previous config saved to /var/cache/conftool/dbconfig/20240221-145407-arnaudb.json |
[production] |
| 14:53 |
<cmooney@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for private1-a-codfw - cmooney@cumin1002" |
[production] |