501-550 of 10000 results (24ms)
2025-07-08 ยง
10:34 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wdqs-all [production]
10:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1159 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78799 and previous config saved to /var/cache/conftool/dbconfig/20250708-103106-marostegui.json [production]
10:31 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1159.eqiad.wmnet with reason: Maintenance [production]
10:27 <marostegui@cumin1002> dbctl commit (dc=all): 'db1159 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78798 and previous config saved to /var/cache/conftool/dbconfig/20250708-102746-root.json [production]
10:27 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet [production]
10:21 <btullis@cumin1003> START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet [production]
10:21 <marostegui@cumin1002> dbctl commit (dc=all): 'db2157 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78797 and previous config saved to /var/cache/conftool/dbconfig/20250708-102140-root.json [production]
10:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1159 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78796 and previous config saved to /var/cache/conftool/dbconfig/20250708-102114-marostegui.json [production]
10:21 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1159.eqiad.wmnet with reason: Maintenance [production]
10:20 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1159.eqiad.wmnet with reason: Maintenance [production]
10:16 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wcqs-public [production]
10:14 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wcqs-public [production]
10:14 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
10:13 <jmm@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad [production]
10:12 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
10:11 <jmm@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad [production]
10:09 <jmm@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw [production]
10:07 <jmm@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw [production]
10:04 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2157', diff saved to https://phabricator.wikimedia.org/P78795 and previous config saved to /var/cache/conftool/dbconfig/20250708-100434-marostegui.json [production]
10:00 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
09:53 <Amir1> dropping term store tables on s8 sanitarium master (T351820) [production]
09:51 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
09:51 <moritzm> installling openssl security updates on Bullseye [production]
09:51 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
09:51 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
09:50 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
09:41 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
09:41 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
09:15 <aklapper@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.9 refs T392179 [production]
09:12 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema-eqiad [production]
09:04 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema-eqiad [production]
08:59 <aklapper@deploy1003> Finished scap sync-world: testwikis to 1.45.0-wmf.9 refs T392179 (duration: 43m 18s) [production]
08:52 <moritzm> installing nginx security updates [production]
08:48 <moritzm> installing Redis security updates [production]
08:30 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [production]
08:30 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [production]
08:30 <moritzm> created a stub user "bumpuid" to move the allocation of UIDs for accounted created in Wikimedia IDM to 100000+ T355663 [production]
08:30 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:28 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:26 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:26 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:16 <aklapper@deploy1003> Started scap sync-world: testwikis to 1.45.0-wmf.9 refs T392179 [production]
08:11 <moritzm> installing postgresql-15 security updates [production]
08:11 <gmodena@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:11 <gmodena@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:06 <gmodena@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:06 <gmodena@deploy1003> helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:02 <gmodena@deploy1003> helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:01 <gmodena@deploy1003> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
07:55 <fabfur> enabling puppet on A:cp (T329332) [production]