|
2023-02-21
ยง
|
| 16:59 |
<bking@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal,name=codfw |
[production] |
| 16:59 |
<bking@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=codfw |
[production] |
| 16:57 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in codfw: None - None |
[production] |
| 16:57 |
<akosiaris@cumin1001> |
START - Cookbook sre.discovery.datacenter status all services in codfw: None - None |
[production] |
| 16:51 |
<nfraison@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1003.eqiad.wmnet with reason: host reimage |
[production] |
| 16:49 |
<akosiaris@cumin1001> |
START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: T327991 - None |
[production] |
| 16:48 |
<nfraison@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1003.eqiad.wmnet with reason: host reimage |
[production] |
| 16:24 |
<bking@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) |
[production] |
| 16:22 |
<papaul> |
rebooting mgmt switch in rack c3 |
[production] |
| 16:20 |
<papaul> |
rebooting mgmt switch in rack b3 |
[production] |
| 16:17 |
<papaul> |
rebooting mgmt switch in rack b1 |
[production] |
| 16:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2108 (T328255)', diff saved to https://phabricator.wikimedia.org/P44704 and previous config saved to /var/cache/conftool/dbconfig/20230221-161552-ladsgroup.json |
[production] |
| 16:15 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 16:15 |
<nfraison@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-presto1003.eqiad.wmnet with OS bullseye |
[production] |
| 16:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance |
[production] |
| 16:13 |
<papaul> |
rebooting mgmt switch in rack a7 |
[production] |
| 16:10 |
<papaul> |
rebooting mgmt switch in rack a5 |
[production] |
| 16:06 |
<moritzm> |
imported libxml2 2.9.4+dfsg1-7+deb10u5+icu67+wmf1 to component/icu67 for buster-wikimedia T329491 |
[production] |
| 16:05 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 16:05 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 16:04 |
<James_F> |
Zuul: [mediawiki/extensions/GrowthExperiments] Add JS doc build/publish jobs T329034 |
[releng] |
| 16:02 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 16:02 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:50 |
<wm-bot2> |
Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus |
[admin] |
| 15:48 |
<wm-bot2> |
Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus |
[admin] |
| 15:29 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.k8s.upgrade-cluster (exit_code=0) Upgrade K8s version: T329664 |
[production] |
| 15:26 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 15:26 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance |
[production] |
| 15:19 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 15:19 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 15:19 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 15:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 15:18 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 15:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 15:17 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 15:17 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 15:17 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 15:17 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 15:17 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2123.codfw.wmnet with reason: Maintenance |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2123.codfw.wmnet with reason: Maintenance |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2111.codfw.wmnet with reason: Maintenance |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2111.codfw.wmnet with reason: Maintenance |
[production] |
| 15:13 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2101.codfw.wmnet with reason: Maintenance |
[production] |
| 15:13 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2101.codfw.wmnet with reason: Maintenance |
[production] |
| 15:07 |
<James_F> |
Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage) |
[releng] |
| 14:36 |
<filippo@cumin1001> |
conftool action : set/pooled=yes; selector: name=thanos-fe2002.codfw.wmnet |
[production] |
| 14:36 |
<filippo@cumin1001> |
conftool action : set/pooled=yes; selector: name=prometheus2005.codfw.wmnet |
[production] |
| 14:29 |
<moritzm> |
installing NSS security updates |
[production] |