|
2023-02-16
ยง
|
| 14:02 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:889762|[simplewiki] Change to 'uca-default-u-kn' category collation (T329815)]] |
[production] |
| 13:52 |
<wm-bot2> |
deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller (7191997) - cookbook ran by taavi@runko |
[tools] |
| 13:47 |
<wm-bot2> |
deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller (7191997) - cookbook ran by taavi@runko |
[toolsbeta] |
| 13:44 |
<wm-bot2> |
build & push docker image docker-registry.tools.wmflabs.org/ingress-admission:1fe8ec4 from https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller (1fe8ec4) - cookbook ran by taavi@runko |
[tools] |
| 13:29 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 13:29 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 13:24 |
<wm-bot2> |
Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 13:24 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 13:23 |
<wm-bot2> |
Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 13:21 |
<wm-bot2> |
Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 13:14 |
<wm-bot2> |
Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 13:14 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 12:58 |
<Rook> |
moving to new cluster. Old one was restarting hub and couldn't find all of its nodes |
[paws] |
| 12:47 |
<wm-bot2> |
build & push docker image docker-registry.tools.wmflabs.org/ingress-admission:e9b9920 from https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller (e9b9920) - cookbook ran by taavi@runko |
[tools] |
| 12:06 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:06 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:05 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:05 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:05 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:05 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:04 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:04 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:03 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:03 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:03 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:03 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 11:39 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 11:39 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 11:38 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 11:38 |
<jayme@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 11:20 |
<wm-bot2> |
Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 11:19 |
<wm-bot2> |
Depooling OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 11:03 |
<wm-bot2> |
Destroying OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 11:01 |
<wm-bot2> |
Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:59 |
<wm-bot2> |
Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:55 |
<claime> |
repool parse1012 for monitoring of possible CPU1 issues |
[production] |
| 10:45 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host idm1001.wikimedia.org with OS bullseye |
[production] |
| 10:37 |
<elukey@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: sync |
[production] |
| 10:36 |
<elukey@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-logging-external: sync |
[production] |
| 10:35 |
<arturo> |
aborrero@tools-k8s-control-1:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml |
[tools] |
| 10:34 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idm1001.wikimedia.org with reason: host reimage |
[production] |
| 10:32 |
<arturo> |
aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml |
[toolsbeta] |
| 10:31 |
<slyngshede@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on idm1001.wikimedia.org with reason: host reimage |
[production] |
| 10:31 |
<jnuche> |
restarted Zuul to clear stale events |
[releng] |
| 10:22 |
<moritzm> |
installing postgresql-11 security updates on maps* |
[production] |
| 10:20 |
<slyngshede@cumin1001> |
START - Cookbook sre.ganeti.reimage for host idm1001.wikimedia.org with OS bullseye |
[production] |
| 10:15 |
<dcaro> |
purges osd daemons 48 and 40 from eqiad ceph cluster (T329709) |
[admin] |
| 10:12 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host idm1001.wikimedia.org |
[production] |
| 10:02 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) idm1001.wikimedia.org on all recursors |
[production] |
| 10:02 |
<slyngshede@cumin1001> |
START - Cookbook sre.dns.wipe-cache idm1001.wikimedia.org on all recursors |
[production] |