1-50 of 10000 results (43ms)
2026-09-01 ยง
11:15 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1004.wikimedia.org with OS bookworm [production]
11:12 <moritzm> installing openjdk-21 security updates [production]
11:12 <jmm@cumin2003> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad [production]
10:59 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1004.wikimedia.org with reason: host reimage [production]
10:53 <jmm@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1004.wikimedia.org with reason: host reimage [production]
10:49 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: Pool back db2902 [production]
10:45 <moritzm> installing Python 3.11 security updates [production]
10:37 <jmm@cumin2003> START - Cookbook sre.hosts.reimage for host urldownloader1004.wikimedia.org with OS bookworm [production]
10:36 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS trixie [production]
10:36 <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5022.eqsin.wmnet on all recursors [production]
10:36 <ayounsi@cumin1003> START - Cookbook sre.dns.wipe-cache cp5022.eqsin.wmnet on all recursors [production]
10:34 <ayounsi@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp5022 - ayounsi@cumin1003" [production]
10:34 <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:34 <ayounsi@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp5022 - ayounsi@cumin1003" [production]
10:30 <ayounsi@cumin1003> START - Cookbook sre.dns.netbox [production]
10:20 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [production]
10:20 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [production]
10:06 <atsuko@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [production]
10:06 <atsuko@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [production]
10:04 <marostegui@cumin1003> START - Cookbook sre.mysql.pool pool db2902: Pool back db2902 [production]
10:04 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [production]
10:04 <marostegui@cumin1003> END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2902: Pool back db2902 [production]
10:04 <marostegui@cumin1003> START - Cookbook sre.mysql.pool pool db2902: Pool back db2902 [production]
10:03 <javiermonton@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [production]
10:03 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: test [production]
10:03 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db2902: test [production]
10:01 <marostegui@cumin1003> END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db2902: test [production]
10:01 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db2902: test [production]
09:58 <marostegui@cumin1003> END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db2902: test [production]
09:58 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db2902: test [production]
09:50 <atsuko@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/echoserver: apply [production]
09:49 <atsuko@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/echoserver: apply [production]
09:48 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
09:48 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
09:48 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
09:47 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
09:38 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:38 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:24 <marostegui@cumin1003> dbctl commit (dc=all): 'Change db1176 and db2230's weight, test-s4 masters, to 0 to mimic the rest of production T427059', diff saved to https://phabricator.wikimedia.org/P96292 and previous config saved to /var/cache/conftool/dbconfig/20260901-092444-marostegui.json [production]
09:02 <marostegui@cumin1003> dbctl commit (dc=all): 'Add db1903 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96291 and previous config saved to /var/cache/conftool/dbconfig/20260901-090233-marostegui.json [production]
09:01 <marostegui@cumin1003> dbctl commit (dc=all): 'Add db1902 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96290 and previous config saved to /var/cache/conftool/dbconfig/20260901-090158-marostegui.json [production]
09:01 <marostegui@cumin1003> dbctl commit (dc=all): 'Add db1901 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96289 and previous config saved to /var/cache/conftool/dbconfig/20260901-090121-marostegui.json [production]
09:00 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage [production]
08:56 <elukey@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage [production]
08:50 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1003.wikimedia.org with OS bookworm [production]
08:47 <filippo@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1074.eqiad.wmnet [production]
08:47 <filippo@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:47 <filippo@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1074.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [production]
08:46 <filippo@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1074.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [production]
08:43 <marostegui@cumin1003> dbctl commit (dc=all): 'Test repool db2902', diff saved to https://phabricator.wikimedia.org/P96288 and previous config saved to /var/cache/conftool/dbconfig/20260901-084317-marostegui.json [production]