5601-5650 of 10000 results (131ms)
2024-09-30 §
08:23 <gmodena@deploy2002> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
08:12 <elukey@cumin1002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-ctrl1003.eqiad.wmnet [production]
08:12 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-ctrl1003.eqiad.wmnet with OS bullseye [production]
08:01 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-ctrl1003.eqiad.wmnet with reason: host reimage [production]
07:57 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-ctrl1003.eqiad.wmnet with reason: host reimage [production]
07:47 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
07:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
07:45 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host aux-k8s-ctrl1003.eqiad.wmnet with OS bullseye [production]
07:33 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002" [production]
07:33 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002" [production]
07:33 <elukey@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-ctrl1003.eqiad.wmnet on all recursors [production]
07:33 <elukey@cumin1002> START - Cookbook sre.dns.wipe-cache aux-k8s-ctrl1003.eqiad.wmnet on all recursors [production]
07:33 <elukey@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:33 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002" [production]
07:33 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002" [production]
07:25 <elukey@cumin1002> START - Cookbook sre.dns.netbox [production]
07:25 <elukey@cumin1002> START - Cookbook sre.ganeti.makevm for new host aux-k8s-ctrl1003.eqiad.wmnet [production]
2024-09-29 §
22:57 <eileen> config revision changed from 88cf5ff4 to a2ba4a8d [production]
2024-09-28 §
22:13 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, T375961] [production]
22:13 <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, T375961] [production]
22:13 <sukhe@cumin1002> END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, no task ID specified] [production]
22:13 <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, no task ID specified] [production]
21:17 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqsin [reason: testing failed, depooling again, no task ID specified] [production]
21:16 <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: depool site eqsin [reason: testing failed, depooling again, no task ID specified] [production]
21:15 <sukhe> sudo cumin 'A:lvs-eqsin' 'systemctl restart pybal' [production]
21:12 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqsin [reason: testing pooling after cr2-eqsin was down and site was depooled, no task ID specified] [production]
21:12 <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: testing pooling after cr2-eqsin was down and site was depooled, no task ID specified] [production]
20:33 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqsin [reason: cr2-eqsin down, no task ID specified] [production]
20:30 <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: depool site eqsin [reason: cr2-eqsin down, no task ID specified] [production]
18:58 <ejegg> disabled CiviCRM contact dedupe job [production]
2024-09-27 §
23:18 <Dreamy_Jazz> Starting time limited scan on enwiki to catch-up with monthly limit rate - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
16:26 <dduvall@deploy2002> Finished scap sync-world: Backport for [[gerrit:1076047|Define $wmgLBFactoryConfigCallback in offline mode]] (duration: 22m 18s) [production]
16:18 <dduvall@deploy2002> dduvall: Continuing with sync [production]
16:17 <dduvall@deploy2002> dduvall: Backport for [[gerrit:1076047|Define $wmgLBFactoryConfigCallback in offline mode]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
16:04 <dduvall@deploy2002> Started scap sync-world: Backport for [[gerrit:1076047|Define $wmgLBFactoryConfigCallback in offline mode]] [production]
13:52 <jayme> imported cricrl 1.31.1-2 to bookworm-wikimedia - T362408 [production]
13:32 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
13:32 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
13:23 <akosiaris@cumin1002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw[1393-1397].eqiad.wmnet [production]
13:22 <akosiaris@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:22 <akosiaris@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1393-1397].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002" [production]
13:21 <akosiaris@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1393-1397].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002" [production]
13:08 <jynus@cumin1002> dbctl commit (dc=all): 'repool es1022 with 100% load', diff saved to https://phabricator.wikimedia.org/P69434 and previous config saved to /var/cache/conftool/dbconfig/20240927-130855-jynus.json [production]
13:08 <akosiaris@cumin1002> START - Cookbook sre.dns.netbox [production]
12:53 <jynus@cumin1002> dbctl commit (dc=all): 'repool es1022 with 1% load', diff saved to https://phabricator.wikimedia.org/P69433 and previous config saved to /var/cache/conftool/dbconfig/20240927-125317-jynus.json [production]
12:48 <akosiaris@cumin1002> START - Cookbook sre.hosts.decommission for hosts mw[1393-1397].eqiad.wmnet [production]
12:47 <akosiaris@cumin1002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw[1374-1393].eqiad.wmnet [production]
12:47 <akosiaris@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:47 <akosiaris@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1374-1393].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002" [production]
12:45 <akosiaris@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1374-1393].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002" [production]