1001-1050 of 10000 results (30ms)
2023-03-13 §
06:22 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance [production]
06:22 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance [production]
06:21 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 138886 [production]
06:19 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 138886 [production]
06:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2100.codfw.wmnet with reason: Maintenance [production]
06:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2100.codfw.wmnet with reason: Maintenance [production]
06:16 <marostegui_> Deploy schema change on s3 codfw dbmaint T329684 [production]
06:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
06:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
04:37 <kart_> Updated cxserver to 2023-03-09-061555-production (T331097, T327102, T326541) [production]
04:19 <kartik@deploy2002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
04:19 <kartik@deploy2002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
04:18 <kartik@deploy2002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
04:17 <kartik@deploy2002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
04:12 <kartik@deploy2002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
04:12 <kartik@deploy2002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
04:10 <Deus> ceres-01: Change datadir to /var/lib/mysql (T329571) [wikisp]
03:07 <Deus> Maintenance on apollo, mars and ceres-01 (T331803) [wikisp]
01:12 <Krinkle> Change codesearch.wmcloud.org proxy from port 3002 (Hound UI) to 3003 (new Codesearch frontend), ref T263354 [codesearch]
2023-03-12 §
23:01 <James_F> Zuul: [mediawiki/extensions/Tabber] Fix archived extension [releng]
23:01 <James_F> Zuul: Add CI for EImage, SearchParserFunction, and StandardWikitext extensions [releng]
22:57 <James_F> Zuul: Add Want to the CI whitelist [releng]
22:50 <Krinkle> Create new https://codesearch-backend.wmcloud.org/ webproxy, points to the same location as codesearch-old. [codesearch]
13:40 <taavi> restart haproxy on tools-k8s-haproxy-3 [tools]
10:47 <elukey> reset offsets on kafka jumbo for benthos webrequest live (as indicated in https://phabricator.wikimedia.org/T331801#8685569) [production]
10:23 <taavi> reload zuul for https://gerrit.wikimedia.org/r/897068 T328240 [releng]
07:50 <elukey> restart benthos-webrequest-live on centrallog1002 - T331801 [production]
07:49 <elukey> restart benthos-webrequest-live on centrallog2002 - T331801 [production]
07:49 <elukey> stop and mask benthos-webrequest-live on centrallog1001 - T331801 [production]
2023-03-11 §
18:38 <wm-bot2> removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
18:36 <wm-bot2> removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
18:34 <wm-bot2> removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
18:31 <taavi> reboot misbehaving tools-sgeexec-10-11 [tools]
01:22 <TheresNoTime> `[samtar@deployment-deploy03 ~]$ scap lock --all --verbose Debugging issue with extension maintenance script, T324233` [releng]
00:20 <mutante> - on phorge1001, enable general query log in mysql (mariadb), to learn about database scheme, don't forget to turn that off so VM doesn't run out of disk (SET GLOBAL general_log=1;) T328595 [devtools]
2023-03-10 §
22:43 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
22:32 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
22:26 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
22:16 <jhathaway@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
21:24 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
21:14 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
21:13 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
21:08 <Krinkle> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896431 [releng]
21:03 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
20:43 <milimetric@deploy2002> Finished deploy [airflow-dags/analytics@4696eff]: Deploying analytics dags from origin/main_airflow_2.5 [airflow-dags@dd7fc78] (duration: 00m 10s) [production]
20:43 <milimetric@deploy2002> Started deploy [airflow-dags/analytics@4696eff]: Deploying analytics dags from origin/main_airflow_2.5 [airflow-dags@dd7fc78] [production]
20:20 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
20:20 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
20:04 <milimetric> deployed refinery with new pageview jobs, patched in a manual copy of static_data/pageview/whitelist/whitelist.tsv because that file was renamed in the most recent version and would have broken jobs otherwise [analytics]
19:39 <milimetric@deploy2002> Finished deploy [analytics/refinery@898a942] (thin): Special deploy for pageview job migration [analytics/refinery@898a942] (duration: 00m 09s) [production]