1-50 of 10000 results (83ms)
2025-04-07 §
14:01 <oblivian@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [production]
14:01 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc1047.eqiad.wmnet [production]
14:00 <oblivian@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-cron: apply [production]
14:00 <oblivian@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [production]
13:59 <oblivian@deploy1003> helmfile [codfw] START helmfile.d/services/mw-cron: apply [production]
13:54 <James_F> Backport window complete. [production]
13:52 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1134309|Improve GeoCrumbs fallback when page property is not (yet) set (T391128)]] (duration: 13m 25s) [production]
13:44 <jforrester@deploy1003> jforrester, cscott: Continuing with sync [production]
13:44 <fabfur> deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/1134689 on A:cp-esams (T384227) [production]
13:44 <jforrester@deploy1003> jforrester, cscott: Backport for [[gerrit:1134309|Improve GeoCrumbs fallback when page property is not (yet) set (T391128)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:43 <fabfur> disable puppet on A:cp-esams to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1134689 (T384227) [production]
13:38 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1134309|Improve GeoCrumbs fallback when page property is not (yet) set (T391128)]] [production]
13:38 <jforrester@deploy1003> Sync cancelled. [production]
13:36 <btullis@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:36 <jforrester@deploy1003> jforrester, cscott: Backport for [[gerrit:1134309|Improve GeoCrumbs fallback when page property is not (yet) set (T391128)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:36 <taavi@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudsw-b1.private.codfw.wikimedia.cloud on codfw recursors [production]
13:36 <taavi@cumin1002> START - Cookbook sre.dns.wipe-cache cloudsw-b1.private.codfw.wikimedia.cloud on codfw recursors [production]
13:36 <sukhe> reprepro -C main include bullseye-wikimedia trafficserver_9.2.9-1wm1_amd64.changes: T390912 [production]
13:35 <btullis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:34 <sukhe> sudo -i reprepro remove bullseye-wikimedia trafficserver: T390912 [production]
13:34 <sukhe> sudo -i reprepro remove bullseye-wikimedia trafficserver [production]
13:32 <sukhe> depool cp4037: reverting to ATS 9.2.9 [production]
13:30 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1134309|Improve GeoCrumbs fallback when page property is not (yet) set (T391128)]] [production]
13:26 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1134106|Shift to Parsoid Fragment support v3 (T390420)]], [[gerrit:1134107|Where Parsoid Read Views are the default, use it for MFE as well (T376048 T374578)]] (duration: 20m 54s) [production]
13:24 <sukhe@cumin1002> END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on P{cp4037.*} and A:cp for 9.2.10-1wm1 [production]
13:21 <sukhe> P{cp4037.*} and A:cp for 9.2.10-1wm1 T390912 [production]
13:21 <sukhe@cumin1002> START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on P{cp4037.*} and A:cp for 9.2.10-1wm1 [production]
13:20 <jelto@deploy1003> helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply [production]
13:20 <jelto@deploy1003> helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [production]
13:20 <jelto@deploy1003> helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [production]
13:19 <jelto@deploy1003> helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [production]
13:19 <jelto@deploy1003> helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [production]
13:19 <jelto@deploy1003> helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [production]
13:18 <jforrester@deploy1003> jforrester, cscott: Continuing with sync [production]
13:14 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2047.codfw.wmnet [production]
13:13 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1046.eqiad.wmnet [production]
13:10 <jforrester@deploy1003> jforrester, cscott: Backport for [[gerrit:1134106|Shift to Parsoid Fragment support v3 (T390420)]], [[gerrit:1134107|Where Parsoid Read Views are the default, use it for MFE as well (T376048 T374578)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:07 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc1046.eqiad.wmnet [production]
13:05 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1134106|Shift to Parsoid Fragment support v3 (T390420)]], [[gerrit:1134107|Where Parsoid Read Views are the default, use it for MFE as well (T376048 T374578)]] [production]
12:56 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2046.codfw.wmnet [production]
12:55 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1045.eqiad.wmnet [production]
12:49 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc2046.codfw.wmnet [production]
12:49 <jiji@cumin1002> START - Cookbook sre.hosts.reboot-single for host mc1045.eqiad.wmnet [production]
12:48 <jelto@deploy1003> helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [production]
12:47 <jelto@deploy1003> helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [production]
12:35 <topranks> cloudsw1-d5-eqiad: add routes for WMCS OpenStack IPv6 aggregate to cloudgw VIP T389958 [production]
12:32 <topranks> cloudsw1-c8-eqiad: add routes for WMCS OpenStack IPv6 aggregate to cloudgw VIP T389958 [production]
11:57 <btullis@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on an-worker1169.eqiad.wmnet with reason: Moving to rack F8 [production]
11:38 <topranks> enable EBGP between cr2-eqiad and cloudsw1-d5-eqiad (IPv6 / cloud vrf) T389958 [production]
11:25 <topranks> enable EBGP between cr1-eqiad and cloudsw1-c8-eqiad (IPv6 / cloud vrf) T389958 [production]