|
2024-06-12
ยง
|
| 17:52 |
<brett> |
authdns-update run on dns1004 (T364891) |
[production] |
| 17:51 |
<brett> |
Repool ulsfo as A:cp-text nvme upgrades are complete (T364891) |
[production] |
| 17:49 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) |
[production] |
| 17:39 |
<brett> |
Remove downtime of cache_text/cp text servers in ulsfo - T364891 |
[production] |
| 17:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2194 (T367261)', diff saved to https://phabricator.wikimedia.org/P64736 and previous config saved to /var/cache/conftool/dbconfig/20240612-173759-marostegui.json |
[production] |
| 17:30 |
<brett@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=cache_text,dc=ulsfo |
[production] |
| 17:26 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 17:25 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 17:25 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 17:25 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 17:24 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 17:24 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 17:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2194 (T367261)', diff saved to https://phabricator.wikimedia.org/P64735 and previous config saved to /var/cache/conftool/dbconfig/20240612-172406-marostegui.json |
[production] |
| 17:24 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance |
[production] |
| 17:23 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance |
[production] |
| 17:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190 (T367261)', diff saved to https://phabricator.wikimedia.org/P64734 and previous config saved to /var/cache/conftool/dbconfig/20240612-172344-marostegui.json |
[production] |
| 17:13 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 17:13 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 17:10 |
<fabfur@cumin1002> |
conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 17:09 |
<fabfur@cumin1002> |
conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 17:09 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:sessionstore |
[production] |
| 17:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P64733 and previous config saved to /var/cache/conftool/dbconfig/20240612-170837-marostegui.json |
[production] |
| 16:56 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:55 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 16:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P64732 and previous config saved to /var/cache/conftool/dbconfig/20240612-165329-marostegui.json |
[production] |
| 16:37 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet) |
[production] |
| 16:31 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 16:28 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 16:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2190 (T367261)', diff saved to https://phabricator.wikimedia.org/P64730 and previous config saved to /var/cache/conftool/dbconfig/20240612-162426-marostegui.json |
[production] |
| 16:24 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance |
[production] |
| 16:24 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance |
[production] |
| 16:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T367261)', diff saved to https://phabricator.wikimedia.org/P64729 and previous config saved to /var/cache/conftool/dbconfig/20240612-162403-marostegui.json |
[production] |
| 16:21 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1182 (T352010)', diff saved to https://phabricator.wikimedia.org/P64728 and previous config saved to /var/cache/conftool/dbconfig/20240612-162134-ladsgroup.json |
[production] |
| 16:21 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:21 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:21 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1156 (T352010)', diff saved to https://phabricator.wikimedia.org/P64727 and previous config saved to /var/cache/conftool/dbconfig/20240612-162110-ladsgroup.json |
[production] |
| 16:20 |
<brett> |
cumin 'A:cp-text and A:ulsfo' 'systemctl poweroff' - T364891 |
[production] |
| 16:19 |
<kamila@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl1003.eqiad.wmnet with OS bullseye |
[production] |
| 16:19 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 8 hosts with reason: T364891 |
[production] |
| 16:18 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on 8 hosts with reason: T364891 |
[production] |
| 16:18 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 16:18 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 16:17 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 16:17 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 16:17 |
<fabfur@cumin1002> |
conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 16:13 |
<fabfur@cumin1002> |
conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 16:11 |
<jhathaway@deploy1002> |
Finished scap: (no justification provided) (duration: 03m 19s) |
[production] |
| 16:10 |
<jhathaway@deploy1002> |
Started scap: (no justification provided) |
[production] |
| 16:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P64726 and previous config saved to /var/cache/conftool/dbconfig/20240612-160856-marostegui.json |
[production] |
| 16:06 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P64725 and previous config saved to /var/cache/conftool/dbconfig/20240612-160603-ladsgroup.json |
[production] |