|
2024-07-16
ยง
|
| 18:14 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.14 refs T366959 |
[production] |
| 18:00 |
<dancy@deploy1002> |
Installing scap version "4.92.0" for 232 hosts |
[production] |
| 17:59 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@f97900c]: Deploy refinery with refinery-source version 0.2.44 for mw on k8s - take 3 [analytics/refinery@f97900c9] (duration: 00m 47s) |
[production] |
| 17:58 |
<otto@deploy1002> |
Started deploy [analytics/refinery@f97900c]: Deploy refinery with refinery-source version 0.2.44 for mw on k8s - take 3 [analytics/refinery@f97900c9] |
[production] |
| 17:58 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@f97900c]: Deploy refinery with refinery-source version 0.2.44 for mw on k8s - take 2 [analytics/refinery@f97900c9] (duration: 02m 44s) |
[production] |
| 17:58 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2156 (T367781)', diff saved to https://phabricator.wikimedia.org/P66672 and previous config saved to /var/cache/conftool/dbconfig/20240716-175820-arnaudb.json |
[production] |
| 17:58 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 17:58 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 17:57 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2156.codfw.wmnet with reason: Maintenance |
[production] |
| 17:57 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2156.codfw.wmnet with reason: Maintenance |
[production] |
| 17:57 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2149 (T367781)', diff saved to https://phabricator.wikimedia.org/P66671 and previous config saved to /var/cache/conftool/dbconfig/20240716-175742-arnaudb.json |
[production] |
| 17:55 |
<otto@deploy1002> |
Started deploy [analytics/refinery@f97900c]: Deploy refinery with refinery-source version 0.2.44 for mw on k8s - take 2 [analytics/refinery@f97900c9] |
[production] |
| 17:55 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@f97900c]: Deploy refinery with refinery-source version 0.2.44 for mw on k8s [analytics/refinery@f97900c9] (duration: 08m 33s) |
[production] |
| 17:55 |
<cdanis@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 17:53 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 17:47 |
<otto@deploy1002> |
Started deploy [analytics/refinery@f97900c]: Deploy refinery with refinery-source version 0.2.44 for mw on k8s [analytics/refinery@f97900c9] |
[production] |
| 17:47 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@f97900c] (hadoop-test): Deploy refinery with refinery-source version 0.2.44 for mw on k8s - TEST [analytics/refinery@f97900c9] (duration: 03m 23s) |
[production] |
| 17:46 |
<swfrench-wmf> |
appservers-rw and api-rw now resolve to failoid - T367949 |
[production] |
| 17:44 |
<otto@deploy1002> |
Started deploy [analytics/refinery@f97900c] (hadoop-test): Deploy refinery with refinery-source version 0.2.44 for mw on k8s - TEST [analytics/refinery@f97900c9] |
[production] |
| 17:44 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=api-rw,name=eqiad [reason: Depooling ahead of turndown - T367949] |
[production] |
| 17:43 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=appservers-rw,name=eqiad [reason: Depooling ahead of turndown - T367949] |
[production] |
| 17:42 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P66670 and previous config saved to /var/cache/conftool/dbconfig/20240716-174235-arnaudb.json |
[production] |
| 17:40 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=api-ro,name=codfw [reason: Depooling ahead of turndown - T367949] |
[production] |
| 17:39 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=appservers-ro,name=codfw [reason: Depooling ahead of turndown - T367949] |
[production] |
| 17:27 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P66669 and previous config saved to /var/cache/conftool/dbconfig/20240716-172727-arnaudb.json |
[production] |
| 17:26 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.ceph.osd.bootstrap_and_add |
[admin] |
| 17:26 |
<wmbot~dcaro@urcuchillay> |
END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) |
[admin] |
| 17:14 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2006.codfw.wmnet with OS bookworm |
[production] |
| 17:14 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 17:12 |
<pt1979@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 17:12 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2149 (T367781)', diff saved to https://phabricator.wikimedia.org/P66668 and previous config saved to /var/cache/conftool/dbconfig/20240716-171220-arnaudb.json |
[production] |
| 17:00 |
<mutante> |
lists2001 - systemctl reset-failed after gerrit:1054610 to fix T370098 |
[production] |
| 16:56 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy2006.codfw.wmnet with reason: host reimage |
[production] |
| 16:56 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.ceph.osd.depool_and_destroy |
[admin] |
| 16:56 |
<wmbot~dcaro@urcuchillay> |
END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) |
[admin] |
| 16:53 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy2006.codfw.wmnet with reason: host reimage |
[production] |
| 16:51 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2149 (T367781)', diff saved to https://phabricator.wikimedia.org/P66667 and previous config saved to /var/cache/conftool/dbconfig/20240716-165135-arnaudb.json |
[production] |
| 16:51 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2149.codfw.wmnet with reason: Maintenance |
[production] |
| 16:51 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2149.codfw.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 100%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66666 and previous config saved to /var/cache/conftool/dbconfig/20240716-164446-arnaudb.json |
[production] |
| 16:44 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1200 (re)pooling @ 100%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66665 and previous config saved to /var/cache/conftool/dbconfig/20240716-164437-arnaudb.json |
[production] |
| 16:44 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1194 (re)pooling @ 100%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66664 and previous config saved to /var/cache/conftool/dbconfig/20240716-164422-arnaudb.json |
[production] |
| 16:39 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2006.codfw.wmnet with OS bookworm |
[production] |
| 16:36 |
<dcaro@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api |
[toolsbeta] |
| 16:36 |
<dcaro@cloudcumin1001> |
START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api |
[toolsbeta] |
| 16:31 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 16:31 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 16:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2127 (T367781)', diff saved to https://phabricator.wikimedia.org/P66663 and previous config saved to /var/cache/conftool/dbconfig/20240716-163059-arnaudb.json |
[production] |
| 16:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1201 (re)pooling @ 75%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66662 and previous config saved to /var/cache/conftool/dbconfig/20240716-162940-arnaudb.json |
[production] |
| 16:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1200 (re)pooling @ 75%: post T365997 repool', diff saved to https://phabricator.wikimedia.org/P66661 and previous config saved to /var/cache/conftool/dbconfig/20240716-162931-arnaudb.json |
[production] |