51-100 of 10000 results (105ms)
2026-04-20 §
14:43 <marostegui@cumin1003> END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1151: after reimage to trixie [production]
14:43 <marostegui@cumin1003> START - Cookbook sre.mysql.pool pool db1151: after reimage to trixie [production]
14:42 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1151.eqiad.wmnet with OS trixie [production]
14:40 <jclark@cumin1003> START - Cookbook sre.hosts.reimage for host dbstore1010.eqiad.wmnet with OS bookworm [production]
14:37 <jclark@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host dbstore1010.eqiad.wmnet with OS bookworm [production]
14:37 <jclark@cumin1003> START - Cookbook sre.hosts.reimage for host dbstore1010.eqiad.wmnet with OS bookworm [production]
14:36 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-be2006.codfw.wmnet with reason: host reimage [production]
14:35 <jclark@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbstore1010.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
14:26 <kevinbazira@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
14:21 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudelastic1012.eqiad.wmnet [production]
14:21 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1166 (T419961)', diff saved to https://phabricator.wikimedia.org/P91215 and previous config saved to /var/cache/conftool/dbconfig/20260420-142120-fceratto.json [production]
14:21 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
14:20 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91214 and previous config saved to /var/cache/conftool/dbconfig/20260420-142050-fceratto.json [production]
14:19 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host cloudelastic1012.eqiad.wmnet [production]
14:19 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1151.eqiad.wmnet with reason: host reimage [production]
14:15 <jclark@cumin1003> START - Cookbook sre.hosts.provision for host dbstore1010.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
14:14 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1151.eqiad.wmnet with reason: host reimage [production]
14:14 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host thanos-be2006.codfw.wmnet with OS bullseye [production]
14:10 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P91213 and previous config saved to /var/cache/conftool/dbconfig/20260420-141042-fceratto.json [production]
14:02 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [production]
14:02 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1253 (T419635)', diff saved to https://phabricator.wikimedia.org/P91212 and previous config saved to /var/cache/conftool/dbconfig/20260420-140203-fceratto.json [production]
14:02 <urandom> upgrade envoyproxy, restbase — T419637 & T410975 [production]
14:00 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1151.eqiad.wmnet with OS trixie [production]
14:00 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P91211 and previous config saved to /var/cache/conftool/dbconfig/20260420-140033-fceratto.json [production]
14:00 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Reimage to Trixie [production]
14:00 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
14:00 <marostegui@cumin1003> START - Cookbook sre.mysql.parsercache [production]
14:00 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db1151: Reimage to Trixie [production]
14:00 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1151.eqiad.wmnet with reason: Reimage to Trixie [production]
14:00 <kevinbazira@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
13:59 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2144.codfw.wmnet,db1151.eqiad.wmnet with reason: Reimage to Trixie [production]
13:52 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2152 (T410589)', diff saved to https://phabricator.wikimedia.org/P91209 and previous config saved to /var/cache/conftool/dbconfig/20260420-135255-ladsgroup.json [production]
13:52 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance [production]
13:51 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P91208 and previous config saved to /var/cache/conftool/dbconfig/20260420-135155-fceratto.json [production]
13:50 <jclark@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:50 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91207 and previous config saved to /var/cache/conftool/dbconfig/20260420-135025-fceratto.json [production]
13:47 <jclark@cumin1003> START - Cookbook sre.dns.netbox [production]
13:45 <jclark@cumin1003> START - Cookbook sre.hosts.provision for host dbstore1010.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
13:44 <jclark@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:44 <jclark@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dbstore1010 to eqiad - jclark@cumin1003" [production]
13:44 <jclark@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dbstore1010 to eqiad - jclark@cumin1003" [production]
13:41 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1157 (T419961)', diff saved to https://phabricator.wikimedia.org/P91206 and previous config saved to /var/cache/conftool/dbconfig/20260420-134158-fceratto.json [production]
13:41 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
13:41 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P91205 and previous config saved to /var/cache/conftool/dbconfig/20260420-134148-fceratto.json [production]
13:37 <jclark@cumin1003> START - Cookbook sre.dns.netbox [production]
13:35 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
13:34 <Lucas_WMDE> UTC afternoon backport+config window done [production]
13:32 <urandom> decommissioning Cassandra, aqs1014 [a,b] — T412830 [production]
13:31 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1253 (T419635)', diff saved to https://phabricator.wikimedia.org/P91204 and previous config saved to /var/cache/conftool/dbconfig/20260420-133139-fceratto.json [production]
13:30 <eevans@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on aqs1014.eqiad.wmnet with reason: Decommissioning — T412830 [production]