|
2023-01-04
ยง
|
| 13:11 |
<btullis@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1001" |
[production] |
| 13:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P42775 and previous config saved to /var/cache/conftool/dbconfig/20230104-130816-marostegui.json |
[production] |
| 13:00 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:874883|Add messages for Gorontalo Wiktionary (gorwiktionary) (T326137)]], [[gerrit:874882|Add messages for Gorontalo Wiktionary (gorwiktionary) (T326137)]] |
[production] |
| 12:58 |
<urbanecm@deploy1002> |
Finished scap: Creating shnwikibooks (T321248) (duration: 07m 38s) |
[production] |
| 12:55 |
<moritzm> |
installing emacs security updates |
[production] |
| 12:54 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 100%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42774 and previous config saved to /var/cache/conftool/dbconfig/20230104-125330-root.json |
[production] |
| 12:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2127 (T326011)', diff saved to https://phabricator.wikimedia.org/P42773 and previous config saved to /var/cache/conftool/dbconfig/20230104-125310-marostegui.json |
[production] |
| 12:51 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:50 |
<urbanecm@deploy1002> |
Started scap: Creating shnwikibooks (T321248) |
[production] |
| 12:48 |
<urbanecm@deploy1002> |
Finished scap: Creating guwwikiquote (T321247) (duration: 07m 44s) |
[production] |
| 12:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2127 (T326011)', diff saved to https://phabricator.wikimedia.org/P42772 and previous config saved to /var/cache/conftool/dbconfig/20230104-124424-marostegui.json |
[production] |
| 12:44 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 12:44 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 12:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T326011)', diff saved to https://phabricator.wikimedia.org/P42771 and previous config saved to /var/cache/conftool/dbconfig/20230104-124403-marostegui.json |
[production] |
| 12:41 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 12:41 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 12:41 |
<urbanecm@deploy1002> |
Started scap: Creating guwwikiquote (T321247) |
[production] |
| 12:40 |
<claime> |
Rolling reboot of api_appserver hosts in codfw paused for https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230104T1200 |
[production] |
| 12:38 |
<urbanecm@deploy1002> |
Finished scap: Creating aswikiquote (T321246) (duration: 07m 49s) |
[production] |
| 12:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 75%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42770 and previous config saved to /var/cache/conftool/dbconfig/20230104-123825-root.json |
[production] |
| 12:35 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.reimage for host cephosd1001.eqiad.wmnet with OS bullseye |
[production] |
| 12:30 |
<urbanecm@deploy1002> |
Started scap: Creating aswikiquote (T321246) |
[production] |
| 12:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P42769 and previous config saved to /var/cache/conftool/dbconfig/20230104-122857-marostegui.json |
[production] |
| 12:27 |
<cgoubert@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) |
[production] |
| 12:26 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:874880|Add namespace translations in Wayuu (T321881)]], [[gerrit:874879|Add namespace translations in Wayuu (T321881)]] (duration: 10m 36s) |
[production] |
| 12:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 50%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42768 and previous config saved to /var/cache/conftool/dbconfig/20230104-122320-root.json |
[production] |
| 12:18 |
<urbanecm@deploy1002> |
urbanecm and urbanecm: Backport for [[gerrit:874880|Add namespace translations in Wayuu (T321881)]], [[gerrit:874879|Add namespace translations in Wayuu (T321881)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 12:16 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:874880|Add namespace translations in Wayuu (T321881)]], [[gerrit:874879|Add namespace translations in Wayuu (T321881)]] |
[production] |
| 12:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P42767 and previous config saved to /var/cache/conftool/dbconfig/20230104-121350-marostegui.json |
[production] |
| 12:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 25%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42766 and previous config saved to /var/cache/conftool/dbconfig/20230104-120815-root.json |
[production] |
| 11:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T326011)', diff saved to https://phabricator.wikimedia.org/P42765 and previous config saved to /var/cache/conftool/dbconfig/20230104-115844-marostegui.json |
[production] |
| 11:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 10%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42764 and previous config saved to /var/cache/conftool/dbconfig/20230104-115310-root.json |
[production] |
| 11:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2109 (T326011)', diff saved to https://phabricator.wikimedia.org/P42763 and previous config saved to /var/cache/conftool/dbconfig/20230104-115011-marostegui.json |
[production] |
| 11:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 11:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 11:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 5%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42761 and previous config saved to /var/cache/conftool/dbconfig/20230104-113805-root.json |
[production] |
| 11:33 |
<jmm@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host puppetdb2003.codfw.wmnet |
[production] |
| 11:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db2151 to dbctl depooled T326206', diff saved to https://phabricator.wikimedia.org/P42759 and previous config saved to /var/cache/conftool/dbconfig/20230104-112801-marostegui.json |
[production] |
| 11:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 1%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42758 and previous config saved to /var/cache/conftool/dbconfig/20230104-112300-root.json |
[production] |
| 11:02 |
<vgutierrez> |
testing HAProxy 2.4.20 in cp4037 and cp4045 |
[production] |
| 10:56 |
<vgutierrez> |
(apt1001) import HAproxy 2.4.20 from third-party repo for buster and bullseye |
[production] |
| 10:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging AKhatun out of all services on: 1098 hosts |
[production] |
| 10:48 |
<jmm@cumin2002> |
START - Cookbook sre.idm.logout Logging AKhatun out of all services on: 1098 hosts |
[production] |
| 10:48 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging AKhatun out of all services on: 894 hosts |
[production] |
| 10:47 |
<jmm@cumin2002> |
START - Cookbook sre.idm.logout Logging AKhatun out of all services on: 894 hosts |
[production] |
| 10:37 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 10:37 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 10:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2124 T326206', diff saved to https://phabricator.wikimedia.org/P42756 and previous config saved to /var/cache/conftool/dbconfig/20230104-103109-marostegui.json |
[production] |
| 10:29 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |