951-1000 of 10000 results (16ms)
2025-07-08 ยง
10:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1159 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78796 and previous config saved to /var/cache/conftool/dbconfig/20250708-102114-marostegui.json [production]
10:21 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1159.eqiad.wmnet with reason: Maintenance [production]
10:20 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1159.eqiad.wmnet with reason: Maintenance [production]
10:16 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wcqs-public [production]
10:14 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wcqs-public [production]
10:14 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
10:13 <jmm@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad [production]
10:12 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
10:11 <jmm@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad [production]
10:09 <jmm@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw [production]
10:07 <jmm@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw [production]
10:04 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2157', diff saved to https://phabricator.wikimedia.org/P78795 and previous config saved to /var/cache/conftool/dbconfig/20250708-100434-marostegui.json [production]
10:00 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
09:53 <Amir1> dropping term store tables on s8 sanitarium master (T351820) [production]
09:51 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
09:51 <moritzm> installling openssl security updates on Bullseye [production]
09:51 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
09:51 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
09:50 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
09:41 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
09:41 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
09:15 <aklapper@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.45.0-wmf.9 refs T392179 [production]
09:12 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema-eqiad [production]
09:04 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema-eqiad [production]
08:59 <aklapper@deploy1003> Finished scap sync-world: testwikis to 1.45.0-wmf.9 refs T392179 (duration: 43m 18s) [production]
08:52 <moritzm> installing nginx security updates [production]
08:48 <moritzm> installing Redis security updates [production]
08:30 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [production]
08:30 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [production]
08:30 <moritzm> created a stub user "bumpuid" to move the allocation of UIDs for accounted created in Wikimedia IDM to 100000+ T355663 [production]
08:30 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:28 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:26 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:26 <gmodena@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [production]
08:16 <aklapper@deploy1003> Started scap sync-world: testwikis to 1.45.0-wmf.9 refs T392179 [production]
08:11 <moritzm> installing postgresql-15 security updates [production]
08:11 <gmodena@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:11 <gmodena@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:06 <gmodena@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:06 <gmodena@deploy1003> helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:02 <gmodena@deploy1003> helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:01 <gmodena@deploy1003> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
07:55 <fabfur> enabling puppet on A:cp (T329332) [production]
07:54 <marostegui> Migrate s3 eqiad to SBR T383795 [production]
07:45 <fabfur> temporary disable puppet on A:cp to apply https://gerrit.wikimedia.org/r/1135643 (T329332) [production]
07:42 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
07:42 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
07:30 <jelto@cumin1003> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade Replica to GitLab 18.0 [production]
07:19 <jelto@cumin1003> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade Replica to GitLab 18.0 [production]
07:14 <tchanders@deploy1003> Finished scap sync-world: Backport for [[gerrit:1166791|temp accounts: Separate digits in user names with hyphens (T381845)]] (duration: 11m 02s) [production]