1001-1050 of 10000 results (108ms)
2024-12-05 ยง
09:55 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance [production]
09:55 <jayme@cumin2002> START - Cookbook sre.hosts.provision for host mw2449.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
09:54 <jayme@cumin2002> START - Cookbook sre.hosts.provision for host mw2448.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
09:48 <jelto@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1045.eqiad.wmnet with OS bookworm [production]
09:47 <jelto@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1044.eqiad.wmnet with OS bookworm [production]
09:40 <jayme@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2450-2451].codfw.wmnet [production]
09:39 <jayme@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host mw[2450-2451].codfw.wmnet [production]
09:39 <jayme@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw[2448-2451].codfw.wmnet with reason: reimage [production]
09:38 <jayme@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw[2448-2451].codfw.wmnet with reason: reimage [production]
09:38 <jayme@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2448-2449].codfw.wmnet [production]
09:37 <jayme@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host mw[2448-2449].codfw.wmnet [production]
09:31 <jelto@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1044.eqiad.wmnet with reason: host reimage [production]
09:25 <jelto@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1044.eqiad.wmnet with reason: host reimage [production]
09:20 <jayme> destroyed unused expiring puppet certs - T381474 [production]
09:15 <fabfur> deploying haproxykafka also on magru and drmrs (T378578) [production]
09:09 <jelto@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1044.eqiad.wmnet with OS bookworm [production]
09:06 <jelto@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1044.eqiad.wmnet wikikube-worker1045.eqiad.wmnet on all recursors [production]
09:06 <jelto@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker1044.eqiad.wmnet wikikube-worker1045.eqiad.wmnet on all recursors [production]
09:04 <jelto@cumin1002> END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1026 to wikikube-worker1045 [production]
09:04 <jelto@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1045 [production]
09:03 <jelto@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1045 [production]
09:03 <jelto@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:03 <jelto@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1026 to wikikube-worker1045 - jelto@cumin1002" [production]
09:03 <jelto@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1026 to wikikube-worker1045 - jelto@cumin1002" [production]
08:58 <jelto@cumin1002> START - Cookbook sre.dns.netbox [production]
08:58 <jelto@cumin1002> START - Cookbook sre.hosts.rename from kubernetes1026 to wikikube-worker1045 [production]
08:58 <jelto@cumin1002> END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1025 to wikikube-worker1044 [production]
08:57 <jelto@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1044 [production]
08:55 <jelto@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1044 [production]
08:55 <jelto@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:55 <jelto@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1025 to wikikube-worker1044 - jelto@cumin1002" [production]
08:54 <jelto@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1025 to wikikube-worker1044 - jelto@cumin1002" [production]
08:49 <jelto@cumin1002> START - Cookbook sre.dns.netbox [production]
08:49 <jelto@cumin1002> START - Cookbook sre.hosts.rename from kubernetes1025 to wikikube-worker1044 [production]
08:47 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
08:46 <moritzm> rebalance Ganeti eqiad/D following server refreshes [production]
08:08 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
08:07 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
08:07 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214 (T371742)', diff saved to https://phabricator.wikimedia.org/P71611 and previous config saved to /var/cache/conftool/dbconfig/20241205-080745-ladsgroup.json [production]
07:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71610 and previous config saved to /var/cache/conftool/dbconfig/20241205-075237-ladsgroup.json [production]
07:37 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71609 and previous config saved to /var/cache/conftool/dbconfig/20241205-073730-ladsgroup.json [production]
07:36 <jelto@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1025-1026].eqiad.wmnet [production]
07:32 <jelto@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1025-1026].eqiad.wmnet [production]
07:22 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214 (T371742)', diff saved to https://phabricator.wikimedia.org/P71608 and previous config saved to /var/cache/conftool/dbconfig/20241205-072223-ladsgroup.json [production]
07:16 <kevinbazira@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
06:31 <marostegui@cumin1002> dbctl commit (dc=all): 'es2043 (re)pooling @ 100%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71607 and previous config saved to /var/cache/conftool/dbconfig/20241205-063132-root.json [production]
06:16 <marostegui@cumin1002> dbctl commit (dc=all): 'es2043 (re)pooling @ 75%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71606 and previous config saved to /var/cache/conftool/dbconfig/20241205-061626-root.json [production]
06:06 <marostegui@cumin1002> dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71605 and previous config saved to /var/cache/conftool/dbconfig/20241205-060631-root.json [production]
06:06 <marostegui@cumin1002> dbctl commit (dc=all): 'es2022 (re)pooling @ 100%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71604 and previous config saved to /var/cache/conftool/dbconfig/20241205-060612-root.json [production]