|
2023-03-13
§
|
| 06:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108 (T329260)', diff saved to https://phabricator.wikimedia.org/P45733 and previous config saved to /var/cache/conftool/dbconfig/20230313-062942-marostegui.json |
[production] |
| 06:29 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 8966 |
[production] |
| 06:27 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 34549 |
[production] |
| 06:27 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 34549 |
[production] |
| 06:25 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 29357 |
[production] |
| 06:25 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 29357 |
[production] |
| 06:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2108 (T329260)', diff saved to https://phabricator.wikimedia.org/P45732 and previous config saved to /var/cache/conftool/dbconfig/20230313-062244-marostegui.json |
[production] |
| 06:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 06:22 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 06:21 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 138886 |
[production] |
| 06:19 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 138886 |
[production] |
| 06:18 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 06:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 06:16 |
<marostegui_> |
Deploy schema change on s3 codfw dbmaint T329684 |
[production] |
| 06:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance |
[production] |
| 06:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance |
[production] |
| 04:37 |
<kart_> |
Updated cxserver to 2023-03-09-061555-production (T331097, T327102, T326541) |
[production] |
| 04:19 |
<kartik@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 04:19 |
<kartik@deploy2002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 04:18 |
<kartik@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 04:17 |
<kartik@deploy2002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 04:12 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 04:12 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 04:10 |
<Deus> |
ceres-01: Change datadir to /var/lib/mysql (T329571) |
[wikisp] |
| 03:07 |
<Deus> |
Maintenance on apollo, mars and ceres-01 (T331803) |
[wikisp] |
| 01:12 |
<Krinkle> |
Change codesearch.wmcloud.org proxy from port 3002 (Hound UI) to 3003 (new Codesearch frontend), ref T263354 |
[codesearch] |
|
2023-03-12
§
|
| 23:01 |
<James_F> |
Zuul: [mediawiki/extensions/Tabber] Fix archived extension |
[releng] |
| 23:01 |
<James_F> |
Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions |
[releng] |
| 22:57 |
<James_F> |
Zuul: Add Want to the CI whitelist |
[releng] |
| 22:50 |
<Krinkle> |
Create new https://codesearch-backend.wmcloud.org/ webproxy, points to the same location as codesearch-old. |
[codesearch] |
| 13:40 |
<taavi> |
restart haproxy on tools-k8s-haproxy-3 |
[tools] |
| 10:47 |
<elukey> |
reset offsets on kafka jumbo for benthos webrequest live (as indicated in https://phabricator.wikimedia.org/T331801#8685569) |
[production] |
| 10:23 |
<taavi> |
reload zuul for https://gerrit.wikimedia.org/r/897068 T328240 |
[releng] |
| 07:50 |
<elukey> |
restart benthos-webrequest-live on centrallog1002 - T331801 |
[production] |
| 07:49 |
<elukey> |
restart benthos-webrequest-live on centrallog2002 - T331801 |
[production] |
| 07:49 |
<elukey> |
stop and mask benthos-webrequest-live on centrallog1001 - T331801 |
[production] |
|
2023-03-11
§
|
| 18:38 |
<wm-bot2> |
removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 18:36 |
<wm-bot2> |
removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 18:34 |
<wm-bot2> |
removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 18:31 |
<taavi> |
reboot misbehaving tools-sgeexec-10-11 |
[tools] |
| 01:22 |
<TheresNoTime> |
`[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233` |
[releng] |
| 00:20 |
<mutante> |
- on phorge1001, enable general query log in mysql (mariadb), to learn about database scheme, don't forget to turn that off so VM doesn't run out of disk (SET GLOBAL general_log=1;) T328595 |
[devtools] |