251-300 of 10000 results (96ms)
2024-11-06 ยง
12:55 <arnaudb@cumin1002> START - Cookbook sre.mysql.depool db2136 - depooling db2136 to clone on db2236 [production]
12:55 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: provisionning db2236.codfw.wmnet - T373579 [production]
12:54 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: provisionning db2236.codfw.wmnet - T373579 [production]
12:54 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: provisionning db2236.codfw.wmnet - T373579 [production]
12:54 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: provisionning db2236.codfw.wmnet - T373579 [production]
12:52 <slyngs> IDP/CAS-SSO Enable Redis TGT backend [production]
12:52 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1014.eqiad.wmnet [production]
12:52 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1014.eqiad.wmnet [production]
12:50 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1001.eqiad.wmnet to drbd [production]
12:41 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1001.eqiad.wmnet to drbd [production]
12:40 <arnaudb@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1206 quickly with 2 steps - test 1087895 [production]
12:25 <arnaudb@cumin1002> START - Cookbook sre.mysql.pool db1206 quickly with 2 steps - test 1087895 [production]
12:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 depool to test cookbook hotfix on CR 1087895', diff saved to https://phabricator.wikimedia.org/P70960 and previous config saved to /var/cache/conftool/dbconfig/20241106-122348-arnaudb.json [production]
12:23 <marostegui> Migrate db1125 to MariaDB 10.6.20 T378940 [production]
12:23 <arnaudb@cumin1002> dbctl commit (dc=all): '"db1206 pending"', diff saved to https://phabricator.wikimedia.org/P70959 and previous config saved to /var/cache/conftool/dbconfig/20241106-122318-arnaudb.json [production]
12:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 on db2230.codfw.wmnet with reason: testing [production]
12:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 6:00:00 on db2230.codfw.wmnet with reason: testing [production]
12:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 on db1125.eqiad.wmnet with reason: testing [production]
12:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 6:00:00 on db1125.eqiad.wmnet with reason: testing [production]
12:09 <arnaudb@cumin1002> END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) db1206 quickly with 2 steps - repool [production]
12:09 <arnaudb@cumin1002> START - Cookbook sre.mysql.pool db1206 quickly with 2 steps - repool [production]
12:06 <mvolz@deploy2002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
12:06 <mvolz@deploy2002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
12:05 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1206', diff saved to https://phabricator.wikimedia.org/P70957 and previous config saved to /var/cache/conftool/dbconfig/20241106-120536-arnaudb.json [production]
12:03 <mvolz@deploy2002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
12:03 <mvolz@deploy2002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
12:02 <mvolz@deploy2002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
12:02 <mvolz@deploy2002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
11:37 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:37 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:32 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:31 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:30 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:30 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:30 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet [production]
11:08 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet [production]
10:50 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2083.codfw.wmnet with OS bullseye [production]
10:43 <fabfur> rolling out haproxykafka on all ULSFO cp hosts (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1087862) (T378578) [production]
10:43 <elukey> depool maps1005 to test an nginx config - T378944 [production]
10:41 <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.2 refs T375661 [production]
10:32 <XioNoX> push new pfw policies - T379127 [production]
10:28 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1001.eqiad.wmnet to plain [production]
10:27 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1001.eqiad.wmnet to plain [production]
10:16 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1014.eqiad.wmnet [production]
10:15 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1014.eqiad.wmnet [production]
10:15 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1014.eqiad.wmnet [production]
10:12 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1014.eqiad.wmnet [production]
10:12 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd1001.eqiad.wmnet to drbd [production]
09:59 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd1001.eqiad.wmnet to drbd [production]
09:59 <jnuche@deploy2002> Finished scap sync-world: Backport for [[gerrit:1087863|Fix automatic category creations by FuzzyBot (T285463)]] (duration: 08m 03s) [production]