|
2023-07-24
ยง
|
| 12:31 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS bullseye |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49655 and previous config saved to /var/cache/conftool/dbconfig/20230724-122536-root.json |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49654 and previous config saved to /var/cache/conftool/dbconfig/20230724-122529-root.json |
[production] |
| 12:17 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['rdb1014.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['rdb1014.eqiad.wmnet'] |
[production] |
| 12:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49653 and previous config saved to /var/cache/conftool/dbconfig/20230724-121653-root.json |
[production] |
| 12:16 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:14 |
<dcausse@deploy1002> |
Finished deploy [airflow-dags/search@e7b9253]: search: fix table name for wmf_raw.mediawiki_page (duration: 00m 12s) |
[production] |
| 12:14 |
<dcausse@deploy1002> |
Started deploy [airflow-dags/search@e7b9253]: search: fix table name for wmf_raw.mediawiki_page |
[production] |
| 12:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1187', diff saved to https://phabricator.wikimedia.org/P49652 and previous config saved to /var/cache/conftool/dbconfig/20230724-121329-root.json |
[production] |
| 12:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49651 and previous config saved to /var/cache/conftool/dbconfig/20230724-121031-root.json |
[production] |
| 12:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49650 and previous config saved to /var/cache/conftool/dbconfig/20230724-121024-root.json |
[production] |
| 12:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2169 (s6, s7)', diff saved to https://phabricator.wikimedia.org/P49649 and previous config saved to /var/cache/conftool/dbconfig/20230724-120609-root.json |
[production] |
| 10:58 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 10:51 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on releases2002.codfw.wmnet,releases1002.eqiad.wmnet with reason: Decommissioning prep |
[production] |
| 10:51 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on releases2002.codfw.wmnet,releases1002.eqiad.wmnet with reason: Decommissioning prep |
[production] |
| 10:48 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-api-int: apply |
[production] |
| 10:47 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:47 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 10:46 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:46 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:45 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:44 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:41 |
<fabfur> |
applying https://gerrit.wikimedia.org/r/c/operations/puppet/+/940880 (T342211) to eqiad DC, only one left (disable keepalive on port 80 on A:cp) |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:39 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcontrol1005 |
[production] |
| 10:39 |
<aborrero@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host cloudcontrol1005 |
[production] |
| 09:31 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1124.eqiad.wmnet onto db1133.eqiad.wmnet |
[production] |
| 09:26 |
<fabfur> |
applying https://gerrit.wikimedia.org/r/c/operations/puppet/+/940873 (T342211) to drmrs DC (disable keepalive on port 80 on A:cp-drmrs) |
[production] |
| 09:26 |
<dcausse@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/rdf-streaming-updater: apply |
[production] |
| 09:24 |
<dcausse@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/rdf-streaming-updater: apply |
[production] |
| 09:22 |
<vgutierrez> |
rollback to trafficserver 9.1.4 in cp4052 - T339134 |
[production] |
| 09:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.mysql.clone of db1124.eqiad.wmnet onto db1133.eqiad.wmnet |
[production] |
| 09:13 |
<dcausse@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 09:12 |
<dcausse@deploy1002> |
helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 09:08 |
<dcausse@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 09:08 |
<dcausse@deploy1002> |
helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 09:03 |
<dcausse@deploy1002> |
helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 09:01 |
<dcausse@deploy1002> |
helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 09:00 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 08:59 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 08:58 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 08:57 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 08:56 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 08:54 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 08:45 |
<vgutierrez> |
testing trafficserver 9.2.1 in cp4052 (upload node) - T339134 |
[production] |