|
2024-02-19
ยง
|
| 10:10 |
<taavi@cumin1002> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::eqiad1::cloudweb |
[production] |
| 10:09 |
<claime> |
restarting thanos-query.service - titan1002 - T356788 |
[production] |
| 10:05 |
<claime> |
restarting thanos-query.service - titan1001 - T356788 |
[production] |
| 10:04 |
<claime> |
restarting thanos-query.service - titan1001 |
[production] |
| 10:02 |
<taavi@cumin1002> |
START - Cookbook sre.puppet.migrate-role for role: wmcs::openstack::eqiad1::cloudweb |
[production] |
| 09:59 |
<taavi@cumin1002> |
conftool action : set/pooled=yes; selector: name=cloudweb1004.wikimedia.org |
[production] |
| 09:55 |
<taavi@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudweb1004.wikimedia.org with OS bullseye |
[production] |
| 09:49 |
<claime> |
Draining mw2442 - failed RAID - T357380 |
[production] |
| 09:27 |
<taavi@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudweb1004.wikimedia.org with reason: host reimage |
[production] |
| 09:24 |
<taavi@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudweb1004.wikimedia.org with reason: host reimage |
[production] |
| 09:12 |
<taavi@cumin1002> |
START - Cookbook sre.hosts.reimage for host cloudweb1004.wikimedia.org with OS bullseye |
[production] |
| 09:10 |
<moritzm> |
installing gnutls28 security updates on bookworm |
[production] |
| 09:06 |
<taavi@cumin1002> |
conftool action : set/pooled=inactive; selector: name=cloudweb1004.wikimedia.org |
[production] |
| 09:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2168 (re)pooling @ 100%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57000 and previous config saved to /var/cache/conftool/dbconfig/20240219-090600-root.json |
[production] |
| 09:01 |
<ladsgroup@deploy2002> |
Finished scap: Backport for [[gerrit:1004614|Set fawiki to read new in pagelinks (T351237)]] (duration: 09m 43s) |
[production] |
| 08:53 |
<ladsgroup@deploy2002> |
ladsgroup: Continuing with sync |
[production] |
| 08:53 |
<ladsgroup@deploy2002> |
ladsgroup: Backport for [[gerrit:1004614|Set fawiki to read new in pagelinks (T351237)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:51 |
<ladsgroup@deploy2002> |
Started scap: Backport for [[gerrit:1004614|Set fawiki to read new in pagelinks (T351237)]] |
[production] |
| 08:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2168 (re)pooling @ 75%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56999 and previous config saved to /var/cache/conftool/dbconfig/20240219-085055-root.json |
[production] |
| 08:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1213 (re)pooling @ 100%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56998 and previous config saved to /var/cache/conftool/dbconfig/20240219-083840-root.json |
[production] |
| 08:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2168 (re)pooling @ 50%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56997 and previous config saved to /var/cache/conftool/dbconfig/20240219-083550-root.json |
[production] |
| 08:34 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply |
[production] |
| 08:33 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply |
[production] |
| 08:25 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.clone of db2166.codfw.wmnet onto db2167.codfw.wmnet |
[production] |
| 08:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1213 (re)pooling @ 75%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56996 and previous config saved to /var/cache/conftool/dbconfig/20240219-082336-root.json |
[production] |
| 08:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2166 T354826', diff saved to https://phabricator.wikimedia.org/P56995 and previous config saved to /var/cache/conftool/dbconfig/20240219-082321-root.json |
[production] |
| 08:23 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 08:22 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 08:22 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance |
[production] |
| 08:22 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance |
[production] |
| 08:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1246 (re)pooling @ 100%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56994 and previous config saved to /var/cache/conftool/dbconfig/20240219-082121-root.json |
[production] |
| 08:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2168 (re)pooling @ 25%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56993 and previous config saved to /var/cache/conftool/dbconfig/20240219-082045-root.json |
[production] |
| 08:19 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2119 (T352010)', diff saved to https://phabricator.wikimedia.org/P56992 and previous config saved to /var/cache/conftool/dbconfig/20240219-081920-ladsgroup.json |
[production] |
| 08:19 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 08:19 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 08:16 |
<moritzm> |
installing runc security updates on buster |
[production] |
| 08:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Place db2167 in s8 T354826', diff saved to https://phabricator.wikimedia.org/P56991 and previous config saved to /var/cache/conftool/dbconfig/20240219-081132-marostegui.json |
[production] |
| 08:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1213 (re)pooling @ 50%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56990 and previous config saved to /var/cache/conftool/dbconfig/20240219-080831-root.json |
[production] |
| 08:07 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Remove db2167 multiinstance', diff saved to https://phabricator.wikimedia.org/P56989 and previous config saved to /var/cache/conftool/dbconfig/20240219-080744-marostegui.json |
[production] |
| 08:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1246 (re)pooling @ 75%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56988 and previous config saved to /var/cache/conftool/dbconfig/20240219-080616-root.json |
[production] |
| 08:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1244 (re)pooling @ 100%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56987 and previous config saved to /var/cache/conftool/dbconfig/20240219-080612-root.json |
[production] |
| 08:05 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2168 (re)pooling @ 10%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56986 and previous config saved to /var/cache/conftool/dbconfig/20240219-080540-root.json |
[production] |
| 08:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2167 T354826', diff saved to https://phabricator.wikimedia.org/P56985 and previous config saved to /var/cache/conftool/dbconfig/20240219-080322-root.json |
[production] |
| 07:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1213 (re)pooling @ 25%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56984 and previous config saved to /var/cache/conftool/dbconfig/20240219-075325-root.json |
[production] |
| 07:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1246 (re)pooling @ 50%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56983 and previous config saved to /var/cache/conftool/dbconfig/20240219-075111-root.json |
[production] |
| 07:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1244 (re)pooling @ 75%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56982 and previous config saved to /var/cache/conftool/dbconfig/20240219-075107-root.json |
[production] |
| 07:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2168 (re)pooling @ 5%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P56981 and previous config saved to /var/cache/conftool/dbconfig/20240219-075035-root.json |
[production] |
| 07:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Place db2168 in s7 T354826', diff saved to https://phabricator.wikimedia.org/P56980 and previous config saved to /var/cache/conftool/dbconfig/20240219-074609-marostegui.json |
[production] |
| 07:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Remove db2168 multiinstance', diff saved to https://phabricator.wikimedia.org/P56979 and previous config saved to /var/cache/conftool/dbconfig/20240219-074450-marostegui.json |
[production] |
| 07:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2168 T354826', diff saved to https://phabricator.wikimedia.org/P56978 and previous config saved to /var/cache/conftool/dbconfig/20240219-074148-root.json |
[production] |