51-100 of 10000 results (47ms)
2024-02-29 ยง
18:13 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2012.codfw.wmnet with reason: Moving lvs2012 primary interface from private1-b-codfw to private1-b2-codfw [production]
18:13 <bd808@deploy2002> helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [production]
18:12 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs2012.codfw.wmnet with reason: Moving lvs2012 primary interface from private1-b-codfw to private1-b2-codfw [production]
18:12 <bd808@deploy2002> helmfile [codfw] START helmfile.d/services/developer-portal: apply [production]
18:12 <bd808@deploy2002> helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [production]
18:12 <bd808@deploy2002> helmfile [eqiad] START helmfile.d/services/developer-portal: apply [production]
18:11 <bd808@deploy2002> helmfile [staging] DONE helmfile.d/services/developer-portal: apply [production]
18:11 <bd808@deploy2002> helmfile [staging] START helmfile.d/services/developer-portal: apply [production]
18:10 <volans@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'sretest1001.eqiad.wmnet$' on ulsfo recursors [production]
18:10 <volans@cumin1002> START - Cookbook sre.dns.wipe-cache 'sretest1001.eqiad.wmnet$' on ulsfo recursors [production]
18:06 <volans@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1001.eqiad.wmnet sretest1002.eqiad.wmnet on all recursors [production]
18:05 <volans@cumin1002> START - Cookbook sre.dns.wipe-cache sretest1001.eqiad.wmnet sretest1002.eqiad.wmnet on all recursors [production]
16:53 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-worker1173.eqiad.wmnet with reason: Investigating disk errors [production]
16:53 <btullis@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-worker1173.eqiad.wmnet with reason: Investigating disk errors [production]
16:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [production]
16:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [production]
16:45 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
16:45 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
16:44 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
16:44 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
16:42 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
16:41 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
16:40 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply [production]
16:40 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/mw-mcrouter: apply [production]
16:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 100%: Excersie over', diff saved to https://phabricator.wikimedia.org/P58262 and previous config saved to /var/cache/conftool/dbconfig/20240229-163459-root.json [production]
16:27 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply [production]
16:26 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/mw-mcrouter: apply [production]
16:19 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 75%: Excersie over', diff saved to https://phabricator.wikimedia.org/P58261 and previous config saved to /var/cache/conftool/dbconfig/20240229-161954-root.json [production]
16:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db2156 (re)pooling @ 100%: After recloning', diff saved to https://phabricator.wikimedia.org/P58260 and previous config saved to /var/cache/conftool/dbconfig/20240229-161629-root.json [production]
16:12 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw [production]
16:12 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw [production]
16:10 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply [production]
16:09 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/mw-mcrouter: apply [production]
16:06 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
16:06 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
16:05 <topranks> Commencing network maintenance migrating servers to new switch codfw rack B7 T355872 [production]
16:04 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 25%: Excersie over', diff saved to https://phabricator.wikimedia.org/P58259 and previous config saved to /var/cache/conftool/dbconfig/20240229-160449-root.json [production]
16:02 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 9 hosts with reason: Migrating servers in codfw rack B7 to lsw1-b7-codfw [production]
16:02 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 0:30:00 on 9 hosts with reason: Migrating servers in codfw rack B7 to lsw1-b7-codfw [production]
16:01 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw-b-codfw,cr[1-2]-codfw,lsw1-b7-codfw with reason: prepping for server uplink migration codfw rack b7 [production]
16:01 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on asw-b-codfw,cr[1-2]-codfw,lsw1-b7-codfw with reason: prepping for server uplink migration codfw rack b7 [production]
16:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db2156 (re)pooling @ 75%: After recloning', diff saved to https://phabricator.wikimedia.org/P58258 and previous config saved to /var/cache/conftool/dbconfig/20240229-160124-root.json [production]
15:59 <topranks> configuring lsw1-b7-codfw in advance of server migration T355872 [production]
15:52 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply [production]
15:52 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/mw-mcrouter: apply [production]
15:49 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 10%: Excersie over', diff saved to https://phabricator.wikimedia.org/P58257 and previous config saved to /var/cache/conftool/dbconfig/20240229-154944-root.json [production]
15:48 <jiji@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply [production]
15:48 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply [production]
15:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db2156 (re)pooling @ 50%: After recloning', diff saved to https://phabricator.wikimedia.org/P58256 and previous config saved to /var/cache/conftool/dbconfig/20240229-154619-root.json [production]
15:46 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/mw-mcrouter: apply [production]