151-200 of 10000 results (56ms)
2022-10-12 §
10:35 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage [production]
10:33 <cgoubert@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=eventstreams,name=codfw [production]
10:33 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2128 (T318955)', diff saved to https://phabricator.wikimedia.org/P35418 and previous config saved to /var/cache/conftool/dbconfig/20221012-103338-ladsgroup.json [production]
10:33 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
10:33 <claime> depooling eventstreams in codfw - T310721 [production]
10:33 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 16:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
10:33 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance [production]
10:33 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2128.codfw.wmnet with reason: Maintenance [production]
10:22 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS bullseye [production]
10:21 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 20115 [production]
10:20 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 20115 [production]
10:01 <cgoubert@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams: apply [production]
10:01 <cgoubert@deploy1002> helmfile [staging] START helmfile.d/services/eventstreams: apply [production]
09:26 <moritzm> draining ganeti1017 T311687 [production]
09:25 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:829561|Replace wordmark/tagline with correct naming style (T307705)]] (duration: 04m 20s) [production]
09:21 <urbanecm@deploy1002> urbanecm and stang: Backport for [[gerrit:829561|Replace wordmark/tagline with correct naming style (T307705)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
09:21 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:829561|Replace wordmark/tagline with correct naming style (T307705)]] [production]
09:12 <jayme> re-enabled puppet on all kubernetes masters (incl. ml & dse) [production]
09:11 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:841187|SVG resources: Run svgo (T320447)]] (duration: 04m 38s) [production]
09:07 <urbanecm@deploy1002> urbanecm and urbanecm: Backport for [[gerrit:841187|SVG resources: Run svgo (T320447)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
09:07 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:841187|SVG resources: Run svgo (T320447)]] [production]
09:05 <jayme> disabling puppet on all kubernetes masters (incl. ml & dse) [production]
08:59 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain [production]
08:58 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to plain [production]
08:54 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd1004.eqiad.wmnet to plain [production]
08:53 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd1004.eqiad.wmnet to plain [production]
08:52 <vgutierrez> partitioning the ATS cache in cp[2033-2034], cp[6003,6011], cp[1081-1082], cp[5004,5010], cp[3056-3057], cp[4024,4028] - T317748 [production]
08:43 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd [production]
08:33 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of dse-k8s-etcd1001.eqiad.wmnet to drbd [production]
08:28 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd1004.eqiad.wmnet to drbd [production]
08:18 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd1004.eqiad.wmnet to drbd [production]
08:17 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of testvm2001.codfw.wmnet to drbd [production]
08:07 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of testvm2001.codfw.wmnet to drbd [production]
08:02 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of testvm2001.codfw.wmnet to plain [production]
08:01 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of testvm2001.codfw.wmnet to plain [production]
07:25 <matthiasmullie> UTC morning backports done [production]
07:25 <mlitn@deploy1002> Finished scap: Backport for [[gerrit:841515|Rescale images based on width alone (T320406)]] (duration: 05m 19s) [production]
07:20 <mlitn@deploy1002> mlitn and mlitn: Backport for [[gerrit:841515|Rescale images based on width alone (T320406)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
07:19 <mlitn@deploy1002> Started scap: Backport for [[gerrit:841515|Rescale images based on width alone (T320406)]] [production]
06:17 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 4826 [production]
06:16 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 4826 [production]
2022-10-11 §
21:36 <mutante> phab1001 / phab2001 - temp. disabled puppet; stopped ssh-phab service; scheduled icinga downtimes for ssh-phab pybal backend alerts - effectively "soft shutting down" the service - T296022 [production]
21:22 <mutante> phab2001 - systemctl stop ssh-phab; temp disable puppet [production]
21:12 <mutante> puppetmaster1001: rm .*.err in /var/run/confd-template [production]
21:10 <mutante> puppetmaster2001: rm .*.err in /var/run/confd-template [production]
20:56 <dzahn@cumin2002> conftool action : set/pooled=yes; selector: dc=codfw,name=phab2001-vcs.codfw.wmnet [production]
20:35 <dzahn@cumin2002> conftool action : set/pooled=yes; selector: dc=eqiad,name=phab1001-vcs.eqiad.wmnet [production]
20:35 <dzahn@cumin2002> conftool action : set/pooled=inactive; selector: dc=codfw,name=phab2001-vcs.codfw.wmnet [production]
20:27 <mutante> depooling git-ssh service backends with confctl - T296022 [production]
20:26 <TheresNoTime> close UTC late backport window [production]