1501-1550 of 10000 results (41ms)
2025-06-04 ยง
11:55 <jmm@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: krb1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" [production]
11:53 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P77040 and previous config saved to /var/cache/conftool/dbconfig/20250604-115357-fceratto.json [production]
11:51 <jmm@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: krb1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" [production]
11:51 <jmm@cumin1003> START - Cookbook sre.hosts.reimage for host ganeti7001.magru.wmnet with OS bookworm [production]
11:51 <samtar@deploy1003> samtar: Continuing with sync [production]
11:47 <samtar@deploy1003> samtar: Backport for [[gerrit:1153581|IS/IS-labs: Enable TemplateDiscovery flags for mediawikiwiki (T377975)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
11:45 <samtar@deploy1003> Started scap sync-world: Backport for [[gerrit:1153581|IS/IS-labs: Enable TemplateDiscovery flags for mediawikiwiki (T377975)]] [production]
11:44 <marostegui@cumin1002> dbctl commit (dc=all): 'db2193 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77038 and previous config saved to /var/cache/conftool/dbconfig/20250604-114430-root.json [production]
11:38 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T395241)', diff saved to https://phabricator.wikimedia.org/P77037 and previous config saved to /var/cache/conftool/dbconfig/20250604-113849-fceratto.json [production]
11:38 <jmm@cumin1003> START - Cookbook sre.dns.netbox [production]
11:35 <mvolz@deploy1003> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
11:35 <bwojtowicz@deploy1003> helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
11:35 <mvolz@deploy1003> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
11:34 <bwojtowicz@deploy1003> helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
11:34 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet [production]
11:34 <jmm@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet [production]
11:33 <mvolz@deploy1003> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
11:32 <mvolz@deploy1003> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
11:32 <jmm@cumin1003> START - Cookbook sre.hosts.decommission for hosts krb1001.eqiad.wmnet [production]
11:31 <mvolz@deploy1003> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
11:31 <mvolz@deploy1003> helmfile [staging] START helmfile.d/services/citoid: apply [production]
11:30 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db1192 (T395241)', diff saved to https://phabricator.wikimedia.org/P77036 and previous config saved to /var/cache/conftool/dbconfig/20250604-113030-fceratto.json [production]
11:30 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
11:30 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T395241)', diff saved to https://phabricator.wikimedia.org/P77035 and previous config saved to /var/cache/conftool/dbconfig/20250604-113005-fceratto.json [production]
11:29 <marostegui@cumin1002> dbctl commit (dc=all): 'db2193 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77034 and previous config saved to /var/cache/conftool/dbconfig/20250604-112923-root.json [production]
11:27 <jmm@cumin1003> START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet [production]
11:25 <mvolz@deploy1003> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
11:25 <mvolz@deploy1003> helmfile [staging] START helmfile.d/services/citoid: apply [production]
11:22 <mvolz@deploy1003> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
11:22 <mvolz@deploy1003> helmfile [staging] START helmfile.d/services/citoid: apply [production]
11:20 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet [production]
11:14 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P77033 and previous config saved to /var/cache/conftool/dbconfig/20250604-111457-fceratto.json [production]
11:14 <claime> Deployed k8s-controller-sidecars version 1.0.2-3 [production]
11:14 <marostegui@cumin1002> dbctl commit (dc=all): 'db2193 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77032 and previous config saved to /var/cache/conftool/dbconfig/20250604-111418-root.json [production]
11:10 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2193.codfw.wmnet with reason: Maintenance [production]
11:09 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2193 T395989', diff saved to https://phabricator.wikimedia.org/P77031 and previous config saved to /var/cache/conftool/dbconfig/20250604-110955-marostegui.json [production]
11:06 <cgoubert@deploy1003> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
11:06 <cgoubert@deploy1003> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
11:06 <cgoubert@deploy1003> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
11:05 <cgoubert@deploy1003> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
11:05 <cgoubert@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:04 <cgoubert@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
11:04 <cgoubert@deploy1003> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
11:03 <cgoubert@deploy1003> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
10:59 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P77030 and previous config saved to /var/cache/conftool/dbconfig/20250604-105950-fceratto.json [production]
10:52 <jmm@cumin1003> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti7001.magru.wmnet to cluster magru03 and group B [production]
10:51 <jmm@cumin1003> START - Cookbook sre.ganeti.addnode for new host ganeti7001.magru.wmnet to cluster magru03 and group B [production]
10:45 <akosiaris@deploy1003> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
10:44 <akosiaris@deploy1003> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:44 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T395241)', diff saved to https://phabricator.wikimedia.org/P77029 and previous config saved to /var/cache/conftool/dbconfig/20250604-104443-fceratto.json [production]