401-450 of 10000 results (91ms)
2024-09-09 §
08:00 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage [production]
07:58 <moritzm> installing openssl security updates [production]
07:52 <arnaudb@cumin1002> dbctl commit (dc=all): 'Remove db2218 from API/vslow/dump T373175', diff saved to https://phabricator.wikimedia.org/P68734 and previous config saved to /var/cache/conftool/dbconfig/20240909-075258-arnaudb.json [production]
07:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Set db2218 with weight 0 T373175', diff saved to https://phabricator.wikimedia.org/P68733 and previous config saved to /var/cache/conftool/dbconfig/20240909-075145-arnaudb.json [production]
07:51 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: Primary switchover s7 T373175 [production]
07:51 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 29 hosts with reason: Primary switchover s7 T373175 [production]
07:39 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bookworm [production]
07:37 <jayme@cumin1002> END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host kubestage2002.codfw.wmnet [production]
07:37 <jayme@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubestage2002.codfw.wmnet with OS bookworm [production]
07:37 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bookworm [production]
07:36 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet [production]
07:33 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet [production]
07:33 <jayme@cumin1002> START - Cookbook sre.k8s.renumber-node Renumbering for host kubestage2002.codfw.wmnet [production]
07:33 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet [production]
07:33 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet [production]
07:17 <moritzm> installing Linux 5.10.223 on bullseye hosts [production]
07:06 <moritzm> roll out debmonitor-client 0.4.0-2+deb11u1 on bullseye hosts [production]
06:56 <moritzm> installing aom security updates [production]
2024-09-07 §
14:44 <ebernhardson@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:44 <ebernhardson@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
14:38 <dani@deploy1003> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
10:28 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues [production]
10:28 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues [production]
2024-09-06 §
22:35 <ebernhardson@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
22:35 <ebernhardson@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:07 <dduvall@deploy1003> Finished deploy [releng/jenkins-deploy@6ca00a7] (releasing): (no justification provided) (duration: 00m 43s) [production]
19:06 <dduvall@deploy1003> Started deploy [releng/jenkins-deploy@6ca00a7] (releasing): (no justification provided) [production]
19:01 <ebernhardson@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:00 <ebernhardson@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
18:57 <ebernhardson@deploy1003> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
18:57 <ebernhardson@deploy1003> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
18:32 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 7:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
18:32 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 7:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
18:12 <brett> Import ncmonitor 1.2.1-1 into bookworm-wikimedia apt archive [production]
17:55 <brett> Import corto 0.3.1-1 into bookworm-wikimedia apt archive [production]
16:46 <kamila_> ran homer on cr*codfw* for T372878 [production]
16:30 <kamila@cumin1002> END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2103.codfw.wmnet [production]
16:30 <kamila@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet [production]
16:30 <kamila@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet [production]