|
2023-07-24
ยง
|
| 13:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49664 and previous config saved to /var/cache/conftool/dbconfig/20230724-131043-root.json |
[production] |
| 13:04 |
<samtar@deploy1002> |
anzx and samtar: Backport for [[gerrit:940464|add citations, concordance, rhymes, reconstruction, therasus, namespaces for mywiktionary (T342516)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 13:03 |
<vgutierrez> |
depooling cp4052 for some ATS 9.2.1 testing - T339134 |
[production] |
| 13:03 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:940464|add citations, concordance, rhymes, reconstruction, therasus, namespaces for mywiktionary (T342516)]] |
[production] |
| 13:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49663 and previous config saved to /var/cache/conftool/dbconfig/20230724-130208-root.json |
[production] |
| 12:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49662 and previous config saved to /var/cache/conftool/dbconfig/20230724-125545-root.json |
[production] |
| 12:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49661 and previous config saved to /var/cache/conftool/dbconfig/20230724-125538-root.json |
[production] |
| 12:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49660 and previous config saved to /var/cache/conftool/dbconfig/20230724-124703-root.json |
[production] |
| 12:40 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 28458 |
[production] |
| 12:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49659 and previous config saved to /var/cache/conftool/dbconfig/20230724-124040-root.json |
[production] |
| 12:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49658 and previous config saved to /var/cache/conftool/dbconfig/20230724-124034-root.json |
[production] |
| 12:40 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 28458 |
[production] |
| 12:36 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host rdb1014.eqiad.wmnet with OS bullseye |
[production] |
| 12:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49656 and previous config saved to /var/cache/conftool/dbconfig/20230724-123158-root.json |
[production] |
| 12:31 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host rdb1013.eqiad.wmnet with OS bullseye |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49655 and previous config saved to /var/cache/conftool/dbconfig/20230724-122536-root.json |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49654 and previous config saved to /var/cache/conftool/dbconfig/20230724-122529-root.json |
[production] |
| 12:17 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['rdb1014.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:17 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['rdb1014.eqiad.wmnet'] |
[production] |
| 12:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49653 and previous config saved to /var/cache/conftool/dbconfig/20230724-121653-root.json |
[production] |
| 12:16 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['rdb1013.eqiad.wmnet'] |
[production] |
| 12:14 |
<dcausse@deploy1002> |
Finished deploy [airflow-dags/search@e7b9253]: search: fix table name for wmf_raw.mediawiki_page (duration: 00m 12s) |
[production] |
| 12:14 |
<dcausse@deploy1002> |
Started deploy [airflow-dags/search@e7b9253]: search: fix table name for wmf_raw.mediawiki_page |
[production] |
| 12:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1187', diff saved to https://phabricator.wikimedia.org/P49652 and previous config saved to /var/cache/conftool/dbconfig/20230724-121329-root.json |
[production] |
| 12:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49651 and previous config saved to /var/cache/conftool/dbconfig/20230724-121031-root.json |
[production] |
| 12:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P49650 and previous config saved to /var/cache/conftool/dbconfig/20230724-121024-root.json |
[production] |
| 12:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2169 (s6, s7)', diff saved to https://phabricator.wikimedia.org/P49649 and previous config saved to /var/cache/conftool/dbconfig/20230724-120609-root.json |
[production] |
| 10:58 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 10:51 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on releases2002.codfw.wmnet,releases1002.eqiad.wmnet with reason: Decommissioning prep |
[production] |
| 10:51 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on releases2002.codfw.wmnet,releases1002.eqiad.wmnet with reason: Decommissioning prep |
[production] |
| 10:48 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-api-int: apply |
[production] |
| 10:47 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:47 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 10:46 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:46 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:45 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:44 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:41 |
<fabfur> |
applying https://gerrit.wikimedia.org/r/c/operations/puppet/+/940880 (T342211) to eqiad DC, only one left (disable keepalive on port 80 on A:cp) |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:39 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcontrol1005 |
[production] |
| 10:39 |
<aborrero@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host cloudcontrol1005 |
[production] |
| 09:31 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1124.eqiad.wmnet onto db1133.eqiad.wmnet |
[production] |
| 09:26 |
<fabfur> |
applying https://gerrit.wikimedia.org/r/c/operations/puppet/+/940873 (T342211) to drmrs DC (disable keepalive on port 80 on A:cp-drmrs) |
[production] |
| 09:26 |
<dcausse@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/rdf-streaming-updater: apply |
[production] |
| 09:24 |
<dcausse@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/rdf-streaming-updater: apply |
[production] |
| 09:22 |
<vgutierrez> |
rollback to trafficserver 9.1.4 in cp4052 - T339134 |
[production] |