|
2022-10-12
ยง
|
| 11:21 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 11:21 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 11:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T318955)', diff saved to https://phabricator.wikimedia.org/P35422 and previous config saved to /var/cache/conftool/dbconfig/20221012-112124-ladsgroup.json |
[production] |
| 11:11 |
<moritzm> |
installing bind9 security updates on buster (client side tools/libs) |
[production] |
| 11:07 |
<jgiannelos@deploy1002> |
Finished deploy [restbase/deploy@0474832]: Update restbase to 1a02cdfb (duration: 25m 48s) |
[production] |
| 11:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P35421 and previous config saved to /var/cache/conftool/dbconfig/20221012-110617-ladsgroup.json |
[production] |
| 11:02 |
<claime> |
repooled eventstreams in codfw - T310721 |
[production] |
| 11:01 |
<cgoubert@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=eventstreams,name=codfw |
[production] |
| 10:58 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventstreams: apply |
[production] |
| 10:58 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventstreams: apply |
[production] |
| 10:57 |
<claime> |
redeploying eventstreams codfw - T310721 |
[production] |
| 10:55 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS bullseye |
[production] |
| 10:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P35420 and previous config saved to /var/cache/conftool/dbconfig/20221012-105111-ladsgroup.json |
[production] |
| 10:49 |
<moritzm> |
installing dbus security updates |
[production] |
| 10:41 |
<jgiannelos@deploy1002> |
Started deploy [restbase/deploy@0474832]: Update restbase to 1a02cdfb |
[production] |
| 10:39 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 10:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T318955)', diff saved to https://phabricator.wikimedia.org/P35419 and previous config saved to /var/cache/conftool/dbconfig/20221012-103604-ladsgroup.json |
[production] |
| 10:35 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 10:33 |
<cgoubert@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=eventstreams,name=codfw |
[production] |
| 10:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2128 (T318955)', diff saved to https://phabricator.wikimedia.org/P35418 and previous config saved to /var/cache/conftool/dbconfig/20221012-103338-ladsgroup.json |
[production] |
| 10:33 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 10:33 |
<claime> |
depooling eventstreams in codfw - T310721 |
[production] |
| 10:33 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 10:33 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 10:33 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 10:29 |
<dcaro> |
deploying new registry-admission controller |
[toolsbeta] |
| 10:22 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS bullseye |
[production] |
| 10:21 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 20115 |
[production] |
| 10:20 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 20115 |
[production] |
| 10:11 |
<wm-bot> |
<valeriobozzolan> overwrite database from source code |
[tools.glams] |
| 10:01 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventstreams: apply |
[production] |
| 10:01 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/eventstreams: apply |
[production] |
| 09:26 |
<moritzm> |
draining ganeti1017 T311687 |
[production] |
| 09:26 |
<wm-bot2> |
cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus |
[tools] |
| 09:25 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:829561|Replace wordmark/tagline with correct naming style (T307705)]] (duration: 04m 20s) |
[production] |
| 09:21 |
<urbanecm@deploy1002> |
urbanecm and stang: Backport for [[gerrit:829561|Replace wordmark/tagline with correct naming style (T307705)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 09:21 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:829561|Replace wordmark/tagline with correct naming style (T307705)]] |
[production] |
| 09:18 |
<wm-bot2> |
cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus |
[tools] |
| 09:12 |
<jayme> |
re-enabled puppet on all kubernetes masters (incl. ml & dse) |
[production] |
| 09:11 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:841187|SVG resources: Run svgo (T320447)]] (duration: 04m 38s) |
[production] |
| 09:07 |
<urbanecm@deploy1002> |
urbanecm and urbanecm: Backport for [[gerrit:841187|SVG resources: Run svgo (T320447)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 09:07 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:841187|SVG resources: Run svgo (T320447)]] |
[production] |
| 09:05 |
<jayme> |
disabling puppet on all kubernetes masters (incl. ml & dse) |
[production] |
| 09:00 |
<dcaro> |
rebooting tools-k8s-control-1 seemed to help, the host is now reachable and joined the k8s cluster correctly |
[tools.tools] |
| 08:59 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain |
[production] |
| 08:58 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain |
[production] |
| 08:56 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 08:54 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd1004.eqiad.wmnet to plain |
[production] |
| 08:53 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd1004.eqiad.wmnet to plain |
[production] |
| 08:52 |
<dcaro> |
rebooting tools-k8s-control-1 as the host is not reachable through ssh and being flagged as down by k8s |
[tools.tools] |