|
2026-02-09
§
|
| 07:26 |
<ayounsi@cumin1003> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aux-k8s-worker1006.eqiad.wmnet with OS bookworm |
[production] |
| 07:26 |
<ayounsi@cumin1003> |
END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host aux-k8s-worker1006 |
[production] |
| 07:25 |
<ayounsi@cumin1003> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 06:23 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2203.codfw.wmnet with reason: Maintenance |
[production] |
| 06:22 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2203.codfw.wmnet with reason: Schema change |
[production] |
| 06:19 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 06:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2203 T416554', diff saved to https://phabricator.wikimedia.org/P88725 and previous config saved to /var/cache/conftool/dbconfig/20260209-061904-marostegui.json |
[production] |
| 06:18 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 06:17 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db2212 to s1 primary and set section read-write T416554', diff saved to https://phabricator.wikimedia.org/P88724 and previous config saved to /var/cache/conftool/dbconfig/20260209-061756-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set s1 codfw as read-only for maintenance - T416554', diff saved to https://phabricator.wikimedia.org/P88723 and previous config saved to /var/cache/conftool/dbconfig/20260209-061732-marostegui.json |
[production] |
| 06:13 |
<marostegui> |
Starting s1 codfw failover from db2203 to db2212 - T416554 |
[production] |
| 06:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db2212 with weight 0 T416554', diff saved to https://phabricator.wikimedia.org/P88722 and previous config saved to /var/cache/conftool/dbconfig/20260209-061218-marostegui.json |
[production] |
| 06:11 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s1 T416554 |
[production] |
| 03:00 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services |
[admin] |
| 02:47 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services |
[admin] |
| 02:47 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment eqiad1 |
[admin] |
| 02:44 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment eqiad1 |
[admin] |
| 02:13 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 12m 52s) |
[production] |
| 02:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
|
2026-02-08
§
|
| 23:47 |
<eileen> |
civicrm upgraded from bc3a8036 to 953cf9f2 |
[fundraising] |
| 20:46 |
<eileen> |
civicrm upgraded from 40189afa to bc3a8036 |
[fundraising] |
| 18:46 |
<wmbot~soda@tools-bastion-15> |
soda built and uploaded a new version |
[tools.yapping-sodium] |
| 16:50 |
<wmbot~lucaswerkmeister@tools-bastion-15> |
deployed c2fa3f188b (refactoring in preparation of SSR, better user agent policy compliance) |
[tools.wdactle] |
| 04:00 |
<wmbot~soda@tools-bastion-15> |
soda built and uploaded a new version |
[tools.yapping-sodium] |
| 02:01 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 01m 01s) |
[production] |
| 02:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
|
2026-02-06
§
|
| 21:00 |
<bd808> |
Added bd808 (self) as co-maintainer to simplify work on T416678 |
[tools.request] |
| 18:09 |
<dzahn@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on zuul2001.codfw.wmnet with reason: WIP |
[production] |
| 18:08 |
<dzahn@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on zuul1001.eqiad.wmnet with reason: WIP |
[production] |
| 17:28 |
<cdobbins@cumin2002> |
END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003*} and A:liberica |
[production] |
| 17:25 |
<cdobbins@cumin2002> |
START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003*} and A:liberica |
[production] |
| 16:45 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply |
[production] |
| 16:44 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply |
[production] |
| 16:40 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply |
[production] |
| 16:39 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply |
[production] |
| 15:41 |
<ejegg> |
fundraising civicrm upgraded from 06842fbf to 40189afa |
[fundraising] |
| 14:57 |
<hashar@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1237489|TypeError: Unsupported operand types: array + null (T416619)]] (duration: 11m 23s) |
[production] |
| 14:53 |
<hashar@deploy2002> |
hashar: Continuing with sync |
[production] |
| 14:50 |
<hashar@deploy2002> |
hashar: Backport for [[gerrit:1237489|TypeError: Unsupported operand types: array + null (T416619)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 14:46 |
<hashar@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1237489|TypeError: Unsupported operand types: array + null (T416619)]] |
[production] |
| 14:42 |
<bking@deploy2002> |
helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 14:42 |
<bking@deploy2002> |
helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 14:39 |
<bking@deploy2002> |
helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 14:39 |
<bking@deploy2002> |
helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 14:39 |
<bking@deploy2002> |
helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 14:38 |
<bking@deploy2002> |
helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |