|
2024-07-16
ยง
|
| 16:14 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 50%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66658 and previous config saved to /var/cache/conftool/dbconfig/20240716-161435-arnaudb.json |
[production] |
| 16:14 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1200 (re)pooling @ 50%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66657 and previous config saved to /var/cache/conftool/dbconfig/20240716-161426-arnaudb.json |
[production] |
| 16:14 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1194 (re)pooling @ 50%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66656 and previous config saved to /var/cache/conftool/dbconfig/20240716-161411-arnaudb.json |
[production] |
| 16:00 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P66655 and previous config saved to /var/cache/conftool/dbconfig/20240716-160044-arnaudb.json |
[production] |
| 15:59 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 25%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66654 and previous config saved to /var/cache/conftool/dbconfig/20240716-155930-arnaudb.json |
[production] |
| 15:59 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1200 (re)pooling @ 25%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66653 and previous config saved to /var/cache/conftool/dbconfig/20240716-155920-arnaudb.json |
[production] |
| 15:59 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1194 (re)pooling @ 25%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66652 and previous config saved to /var/cache/conftool/dbconfig/20240716-155905-arnaudb.json |
[production] |
| 15:58 |
<elukey> |
uploaded spicerack_8.7.0 to apt.wikimedia.org bullseye-wikimedia |
[production] |
| 15:52 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P66651 and previous config saved to /var/cache/conftool/dbconfig/20240716-155221-root.json |
[production] |
| 15:45 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2127 (T367781)', diff saved to https://phabricator.wikimedia.org/P66650 and previous config saved to /var/cache/conftool/dbconfig/20240716-154537-arnaudb.json |
[production] |
| 15:44 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 10%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66649 and previous config saved to /var/cache/conftool/dbconfig/20240716-154424-arnaudb.json |
[production] |
| 15:44 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1200 (re)pooling @ 10%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66648 and previous config saved to /var/cache/conftool/dbconfig/20240716-154415-arnaudb.json |
[production] |
| 15:44 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1194 (re)pooling @ 10%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66647 and previous config saved to /var/cache/conftool/dbconfig/20240716-154401-arnaudb.json |
[production] |
| 15:39 |
<dcausse@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:39 |
<dcausse@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:37 |
<papaul> |
reboot fpc0 on fasw-c-codfw.mgmt.codfw.wmnet |
[production] |
| 15:37 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P66646 and previous config saved to /var/cache/conftool/dbconfig/20240716-153715-root.json |
[production] |
| 15:36 |
<dcausse@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:35 |
<dcausse@deploy1002> |
helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:32 |
<dcausse@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:32 |
<dcausse@deploy1002> |
helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 5%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66645 and previous config saved to /var/cache/conftool/dbconfig/20240716-152918-arnaudb.json |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1200 (re)pooling @ 5%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66644 and previous config saved to /var/cache/conftool/dbconfig/20240716-152910-arnaudb.json |
[production] |
| 15:28 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1194 (re)pooling @ 5%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66643 and previous config saved to /var/cache/conftool/dbconfig/20240716-152855-arnaudb.json |
[production] |
| 15:27 |
<cgoubert@cumin1002> |
conftool action : set/pooled=yes; selector: name=(kubernetes1062.eqiad.wmnet|mw1494.eqiad.wmnet|mw1495.eqiad.wmnet),cluster=kubernetes,service=kubesvc |
[production] |
| 15:27 |
<claime> |
Uncordoning kubernetes1062.eqiad.wmnet mw1494.eqiad.wmnet mw1495.eqiad.wmnet - T365997 |
[production] |
| 15:23 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2127 (T367781)', diff saved to https://phabricator.wikimedia.org/P66642 and previous config saved to /var/cache/conftool/dbconfig/20240716-152349-arnaudb.json |
[production] |
| 15:23 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 15:23 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 15:22 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1158 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P66641 and previous config saved to /var/cache/conftool/dbconfig/20240716-152209-root.json |
[production] |
| 15:19 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:19 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:15 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1240.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:15 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1240.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:15 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1212 (T367781)', diff saved to https://phabricator.wikimedia.org/P66640 and previous config saved to /var/cache/conftool/dbconfig/20240716-151516-arnaudb.json |
[production] |
| 15:08 |
<topranks> |
Rebooting lsw1-f2-eqiad to complete JunOS upgrade T365997 |
[production] |
| 15:08 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 21 hosts with reason: JunOS upgrade lsw1-f2-eqiad |
[production] |
| 15:07 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on 21 hosts with reason: JunOS upgrade lsw1-f2-eqiad |
[production] |
| 15:07 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lsw1-f2-eqiad,lsw1-f2-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-f2-eqiad |
[production] |
| 15:07 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1158 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P66638 and previous config saved to /var/cache/conftool/dbconfig/20240716-150704-root.json |
[production] |
| 15:06 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on lsw1-f2-eqiad,lsw1-f2-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-f2-eqiad |
[production] |
| 15:06 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@7335128]: deploy phab1004 for T370109 (duration: 00m 52s) |
[production] |
| 15:05 |
<godog> |
silence OtelCollectorRefusedSpans in codfw for 7d - T370043 |
[production] |
| 15:05 |
<godog> |
silence OtelCollectorRefusedSpans in codfw for 7d |
[production] |
| 15:05 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@7335128]: deploy phab1004 for T370109 |
[production] |
| 15:04 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@7335128]: test deploy phab2002 for T370109 (duration: 00m 34s) |
[production] |
| 15:04 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab.wmfusercontent.org with reason: Phabricator/Phorge update |
[production] |
| 15:04 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab.wmfusercontent.org with reason: Phabricator/Phorge update |
[production] |
| 15:04 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@7335128]: test deploy phab2002 for T370109 |
[production] |
| 15:02 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |