151-200 of 10000 results (99ms)
2025-07-11 ยง
10:00 <marostegui@cumin1002> dbctl commit (dc=all): 'db2192 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78906 and previous config saved to /var/cache/conftool/dbconfig/20250711-100033-root.json [production]
09:45 <marostegui@cumin1002> dbctl commit (dc=all): 'db2192 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78905 and previous config saved to /var/cache/conftool/dbconfig/20250711-094527-root.json [production]
09:39 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2192.codfw.wmnet with reason: Maintenance [production]
09:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2192 T399280', diff saved to https://phabricator.wikimedia.org/P78904 and previous config saved to /var/cache/conftool/dbconfig/20250711-093115-root.json [production]
09:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2213 to s5 primary T399280', diff saved to https://phabricator.wikimedia.org/P78903 and previous config saved to /var/cache/conftool/dbconfig/20250711-093006-marostegui.json [production]
09:29 <marostegui> Starting s5 codfw failover from db2192 to db2213 - T399280 [production]
09:27 <jmm@cumin1003> START - Cookbook sre.hosts.reimage for host sretest1003.eqiad.wmnet with OS trixie [production]
09:25 <moritzm> imported perccli for trixie-wikimedia T391083 [production]
09:18 <marostegui@cumin1002> dbctl commit (dc=all): 'Remove db2213 from API/vslow/dump T399280', diff saved to https://phabricator.wikimedia.org/P78902 and previous config saved to /var/cache/conftool/dbconfig/20250711-091812-root.json [production]
09:15 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 T399280 [production]
09:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db2223 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P78901 and previous config saved to /var/cache/conftool/dbconfig/20250711-091242-root.json [production]
09:04 <jmm@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1003.eqiad.wmnet with OS trixie [production]
08:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db2223 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P78900 and previous config saved to /var/cache/conftool/dbconfig/20250711-085736-root.json [production]
08:51 <elukey@deploy1003> helmfile [codfw] DONE helmfile.d/admin 'sync'. [production]
08:51 <elukey@deploy1003> helmfile [codfw] START helmfile.d/admin 'sync'. [production]
08:51 <elukey@deploy1003> helmfile [eqiad] DONE helmfile.d/admin 'sync'. [production]
08:51 <elukey@deploy1003> helmfile [eqiad] START helmfile.d/admin 'sync'. [production]
08:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db2223 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78899 and previous config saved to /var/cache/conftool/dbconfig/20250711-084230-root.json [production]
08:42 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/changeprop: sync [production]
08:41 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/changeprop: sync [production]
08:34 <jmm@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts puppetserver2003.codfw.wmnet [production]
08:34 <jmm@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:34 <jmm@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetserver2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" [production]
08:33 <jmm@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetserver2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" [production]
08:30 <jmm@cumin1003> START - Cookbook sre.dns.netbox [production]
08:27 <marostegui@cumin1002> dbctl commit (dc=all): 'db2223 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78898 and previous config saved to /var/cache/conftool/dbconfig/20250711-082725-root.json [production]
08:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2223 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78897 and previous config saved to /var/cache/conftool/dbconfig/20250711-081953-marostegui.json [production]
08:19 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2223.codfw.wmnet with reason: Maintenance [production]
08:17 <jmm@cumin1003> START - Cookbook sre.hosts.decommission for hosts puppetserver2003.codfw.wmnet [production]
08:02 <jmm@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1003.eqiad.wmnet with reason: host reimage [production]
07:56 <jmm@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1003.eqiad.wmnet with reason: host reimage [production]
07:36 <jmm@cumin1003> START - Cookbook sre.hosts.reimage for host sretest1003.eqiad.wmnet with OS trixie [production]
07:24 <marostegui@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P78896 and previous config saved to /var/cache/conftool/dbconfig/20250711-072439-root.json [production]
07:09 <marostegui@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P78895 and previous config saved to /var/cache/conftool/dbconfig/20250711-070933-root.json [production]
06:54 <marostegui@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P78894 and previous config saved to /var/cache/conftool/dbconfig/20250711-065428-root.json [production]
06:39 <marostegui@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P78893 and previous config saved to /var/cache/conftool/dbconfig/20250711-063922-root.json [production]
06:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2213 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P78892 and previous config saved to /var/cache/conftool/dbconfig/20250711-063156-marostegui.json [production]
06:31 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2213.codfw.wmnet with reason: Maintenance [production]
03:35 <andrew@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts cloudcephosd1041.eqiad.wmnet [production]
03:35 <andrew@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1041.eqiad.wmnet [production]
03:21 <andrew@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1041.eqiad.wmnet [production]
03:13 <andrew@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1041.eqiad.wmnet [production]
02:30 <andrew@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts cloudcephosd1040.eqiad.wmnet [production]
02:30 <andrew@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1040.eqiad.wmnet [production]
02:17 <andrew@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudcephosd1040.eqiad.wmnet [production]
02:13 <andrew@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1040.eqiad.wmnet [production]
02:11 <andrew@cumin2002> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cloudcephosd1039.eqiad.wmnet [production]
02:11 <andrew@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1039.eqiad.wmnet [production]
02:10 <andrew@cumin2002> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cloudcephosd1039.eqiad.wmnet [production]
02:10 <andrew@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1039.eqiad.wmnet [production]