|
2024-09-10
ยง
|
| 15:48 |
<bking@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bullseye |
[production] |
| 15:47 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68814 and previous config saved to /var/cache/conftool/dbconfig/20240910-154740-arnaudb.json |
[production] |
| 15:46 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: network maintenance T373097 |
[production] |
| 15:45 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: network maintenance T373097 |
[production] |
| 15:45 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'depool db2126 db2165 db2166 db2192 db2208 es2037 - T370852', diff saved to https://phabricator.wikimedia.org/P68813 and previous config saved to /var/cache/conftool/dbconfig/20240910-154540-arnaudb.json |
[production] |
| 15:39 |
<mutante> |
enabling throttling on Gerrit hosts - T365259 |
[production] |
| 15:39 |
<jelto> |
enabling throttling on GitLab hosts - T366882 |
[production] |
| 15:34 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2012.codfw.wmnet |
[production] |
| 15:34 |
<cgoubert@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID |
[production] |
| 15:34 |
<cgoubert@cumin1002> |
START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID |
[production] |
| 15:33 |
<cgoubert@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet |
[production] |
| 15:33 |
<cgoubert@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet |
[production] |
| 15:32 |
<cmooney@cumin1002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2012.codfw.wmnet |
[production] |
| 15:32 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2127 (re)pooling @ 15%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68811 and previous config saved to /var/cache/conftool/dbconfig/20240910-153234-arnaudb.json |
[production] |
| 15:17 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2127 (re)pooling @ 5%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68810 and previous config saved to /var/cache/conftool/dbconfig/20240910-151729-arnaudb.json |
[production] |
| 15:16 |
<ebernhardson@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:16 |
<ebernhardson@deploy1003> |
helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:12 |
<ebernhardson@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:12 |
<ebernhardson@deploy1003> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:08 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=dns2005.wikimedia.org [reason: T373097 codfw maintenance] |
[production] |
| 15:06 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2205.codfw.wmnet with reason: T374425 |
[production] |
| 15:06 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2205.codfw.wmnet with reason: T374425 |
[production] |
| 15:05 |
<brennen@deploy1003> |
Finished deploy [phabricator/deployment@84ada67]: deploy phab1004 for T374458 (duration: 00m 50s) |
[production] |
| 15:04 |
<brennen@deploy1003> |
Started deploy [phabricator/deployment@84ada67]: deploy phab1004 for T374458 |
[production] |
| 15:04 |
<brennen@deploy1003> |
Finished deploy [phabricator/deployment@84ada67]: deploy phab2002 for T374458 (duration: 00m 36s) |
[production] |
| 15:03 |
<brennen@deploy1003> |
Started deploy [phabricator/deployment@84ada67]: deploy phab2002 for T374458 |
[production] |
| 15:02 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:02 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:02 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:02 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2127 (re)pooling @ 4%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68809 and previous config saved to /var/cache/conftool/dbconfig/20240910-150222-arnaudb.json |
[production] |
| 15:02 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:02 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2011.codfw.wmnet |
[production] |
| 15:02 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab.wmfusercontent.org with reason: Phabricator/Phorge update |
[production] |
| 15:01 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab.wmfusercontent.org with reason: Phabricator/Phorge update |
[production] |
| 14:57 |
<cmooney@cumin1002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2011.codfw.wmnet |
[production] |
| 14:47 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1186 (T371742)', diff saved to https://phabricator.wikimedia.org/P68808 and previous config saved to /var/cache/conftool/dbconfig/20240910-144725-ladsgroup.json |
[production] |
| 14:47 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:47 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2127 (re)pooling @ 3%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68807 and previous config saved to /var/cache/conftool/dbconfig/20240910-144716-arnaudb.json |
[production] |
| 14:47 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:47 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T371742)', diff saved to https://phabricator.wikimedia.org/P68806 and previous config saved to /var/cache/conftool/dbconfig/20240910-144703-ladsgroup.json |
[production] |
| 14:37 |
<sukhe> |
sudo cumin -b11 "A:cp" 'run-puppet-agent --enable "merging CRs 1065286"' |
[production] |
| 14:35 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase2033.codfw.wmnet |
[production] |
| 14:35 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase2025.codfw.wmnet |
[production] |
| 14:32 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2127 (re)pooling @ 2%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68805 and previous config saved to /var/cache/conftool/dbconfig/20240910-143211-arnaudb.json |
[production] |
| 14:31 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P68804 and previous config saved to /var/cache/conftool/dbconfig/20240910-143156-ladsgroup.json |
[production] |
| 14:29 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 14:28 |
<sukhe> |
[end] rolling restart of {pdns-recursor,haproxy}.service on A:dnsbox |
[production] |
| 14:25 |
<jayme> |
restoring leadership for partitions assigned to broker id 2002 on kafka-main-codfw - T363210 |
[production] |
| 14:17 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2127 (re)pooling @ 1%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68802 and previous config saved to /var/cache/conftool/dbconfig/20240910-141705-arnaudb.json |
[production] |
| 14:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P68801 and previous config saved to /var/cache/conftool/dbconfig/20240910-141649-ladsgroup.json |
[production] |