1001-1050 of 10000 results (90ms)
2024-02-22 ยง
14:29 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
14:29 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
14:29 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T357189)', diff saved to https://phabricator.wikimedia.org/P57722 and previous config saved to /var/cache/conftool/dbconfig/20240222-142859-arnaudb.json [production]
14:28 <cgoubert@deploy2002> Started scap: Backport for [[gerrit:1004135|Enable $wgLocalHTTPProxy on group1 wikis (T298265)]] [production]
14:15 <marostegui@cumin1002> dbctl commit (dc=all): 'es1028 (re)pooling @ 100%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57721 and previous config saved to /var/cache/conftool/dbconfig/20240222-141508-root.json [production]
14:13 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P57720 and previous config saved to /var/cache/conftool/dbconfig/20240222-141353-arnaudb.json [production]
14:03 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
14:00 <marostegui@cumin1002> dbctl commit (dc=all): 'es1028 (re)pooling @ 75%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57719 and previous config saved to /var/cache/conftool/dbconfig/20240222-140003-root.json [production]
13:58 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P57718 and previous config saved to /var/cache/conftool/dbconfig/20240222-135846-arnaudb.json [production]
13:53 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:52 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
13:52 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:52 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
13:52 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:52 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
13:52 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:51 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
13:51 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:51 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
13:51 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:46 <aborrero@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1034.eqiad.wmnet with OS bookworm [production]
13:46 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
13:45 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:45 <btullis@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [production]
13:45 <marostegui@cumin1002> dbctl commit (dc=all): 'es1028 (re)pooling @ 50%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57717 and previous config saved to /var/cache/conftool/dbconfig/20240222-134458-root.json [production]
13:44 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T357189)', diff saved to https://phabricator.wikimedia.org/P57716 and previous config saved to /var/cache/conftool/dbconfig/20240222-134340-arnaudb.json [production]
13:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1193 (T357189)', diff saved to https://phabricator.wikimedia.org/P57715 and previous config saved to /var/cache/conftool/dbconfig/20240222-134120-arnaudb.json [production]
13:41 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:41 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
13:41 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
13:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T357189)', diff saved to https://phabricator.wikimedia.org/P57714 and previous config saved to /var/cache/conftool/dbconfig/20240222-134059-arnaudb.json [production]
13:40 <aborrero@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1034 [production]
13:40 <aborrero@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1034 [production]
13:34 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:29 <marostegui@cumin1002> dbctl commit (dc=all): 'es1028 (re)pooling @ 25%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57713 and previous config saved to /var/cache/conftool/dbconfig/20240222-132953-root.json [production]
13:25 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P57712 and previous config saved to /var/cache/conftool/dbconfig/20240222-132551-arnaudb.json [production]
13:20 <btullis@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [production]
13:20 <aborrero@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1034.eqiad.wmnet with reason: host reimage [production]
13:18 <aborrero@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1034.eqiad.wmnet with reason: host reimage [production]
13:14 <marostegui@cumin1002> dbctl commit (dc=all): 'es1028 (re)pooling @ 10%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57711 and previous config saved to /var/cache/conftool/dbconfig/20240222-131448-root.json [production]
13:13 <godog> bounce grafana to apply new datasources [production]
13:10 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P57710 and previous config saved to /var/cache/conftool/dbconfig/20240222-131045-arnaudb.json [production]
13:05 <Emperor> ms-codfw set ACL {"read-only":["mw:backup"]} T269108 [production]
13:03 <Emperor> ms-eqiad set ACL {"read-only":["mw:backup"]} T269108 [production]
13:02 <eoghan@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrading gitlab [production]
13:01 <aborrero@cumin1002> START - Cookbook sre.hosts.reimage for host cloudvirt1034.eqiad.wmnet with OS bookworm [production]
12:59 <marostegui@cumin1002> dbctl commit (dc=all): 'es1028 (re)pooling @ 5%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57709 and previous config saved to /var/cache/conftool/dbconfig/20240222-125943-root.json [production]
12:55 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T357189)', diff saved to https://phabricator.wikimedia.org/P57708 and previous config saved to /var/cache/conftool/dbconfig/20240222-125538-arnaudb.json [production]
12:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1192 (T357189)', diff saved to https://phabricator.wikimedia.org/P57707 and previous config saved to /var/cache/conftool/dbconfig/20240222-125319-arnaudb.json [production]