|
2024-06-04
§
|
| 07:42 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS bookworm |
[production] |
| 07:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T364299)', diff saved to https://phabricator.wikimedia.org/P63983 and previous config saved to /var/cache/conftool/dbconfig/20240604-073830-marostegui.json |
[production] |
| 07:27 |
<marostegui> |
dbmaint eqiad s1 deploy schema change on db1184 T356166 |
[production] |
| 07:15 |
<moritzm> |
installing intel-microcode updates on bullseye |
[production] |
| 07:10 |
<marostegui> |
dbmaint eqiad s1 deploy schema change on db1184 T355609 |
[production] |
| 07:06 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:06 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:05 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS bookworm |
[production] |
| 06:43 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage |
[production] |
| 06:40 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage |
[production] |
| 06:26 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS bookworm |
[production] |
| 06:26 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1184.eqiad.wmnet with reason: reimage |
[production] |
| 06:26 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on db1184.eqiad.wmnet with reason: reimage |
[production] |
| 06:14 |
<marostegui> |
Rename table flaggedpage_pending on db1185 (s5 eqiad dbmaint) - T365568 |
[production] |
| 06:09 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): ' fix api db1163 vs db1184 T366259', diff saved to https://phabricator.wikimedia.org/P63982 and previous config saved to /var/cache/conftool/dbconfig/20240604-060925-arnaudb.json |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'API db1163 T366259', diff saved to https://phabricator.wikimedia.org/P63981 and previous config saved to /var/cache/conftool/dbconfig/20240604-060747-arnaudb.json |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depool db1184 T366259', diff saved to https://phabricator.wikimedia.org/P63980 and previous config saved to /var/cache/conftool/dbconfig/20240604-060703-arnaudb.json |
[production] |
| 06:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Promote db1163 to s1 primary and set section read-write T366259', diff saved to https://phabricator.wikimedia.org/P63979 and previous config saved to /var/cache/conftool/dbconfig/20240604-060324-arnaudb.json |
[production] |
| 06:02 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T366259', diff saved to https://phabricator.wikimedia.org/P63978 and previous config saved to /var/cache/conftool/dbconfig/20240604-060208-arnaudb.json |
[production] |
| 06:01 |
<arnaudb> |
Starting s1 eqiad failover from db1184 to db1163 - T366259 |
[production] |
| 05:28 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Set db1163 with weight 0 T366259', diff saved to https://phabricator.wikimedia.org/P63977 and previous config saved to /var/cache/conftool/dbconfig/20240604-052803-arnaudb.json |
[production] |
| 05:27 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 35 hosts with reason: Primary switchover s1 T366259 |
[production] |
| 05:27 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 35 hosts with reason: Primary switchover s1 T366259 |
[production] |
| 04:20 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1246 (T352010)', diff saved to https://phabricator.wikimedia.org/P63976 and previous config saved to /var/cache/conftool/dbconfig/20240604-042011-ladsgroup.json |
[production] |
| 04:20 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:19 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:01 |
<mwpresync@deploy1002> |
Pruned MediaWiki: 1.43.0-wmf.5 (duration: 00m 57s) |
[production] |
| 03:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2216 (T364299)', diff saved to https://phabricator.wikimedia.org/P63975 and previous config saved to /var/cache/conftool/dbconfig/20240604-035703-marostegui.json |
[production] |
| 03:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2216.codfw.wmnet with reason: Maintenance |
[production] |
| 03:56 |
<mwpresync@deploy1002> |
Finished scap: testwikis wikis to 1.43.0-wmf.8 refs T361402 (duration: 53m 47s) |
[production] |
| 03:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2216.codfw.wmnet with reason: Maintenance |
[production] |
| 03:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2212 (T364299)', diff saved to https://phabricator.wikimedia.org/P63974 and previous config saved to /var/cache/conftool/dbconfig/20240604-035640-marostegui.json |
[production] |
| 03:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P63973 and previous config saved to /var/cache/conftool/dbconfig/20240604-034132-marostegui.json |
[production] |
| 03:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P63972 and previous config saved to /var/cache/conftool/dbconfig/20240604-032625-marostegui.json |
[production] |
| 03:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2212 (T364299)', diff saved to https://phabricator.wikimedia.org/P63971 and previous config saved to /var/cache/conftool/dbconfig/20240604-031117-marostegui.json |
[production] |
| 03:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2212 (T364299)', diff saved to https://phabricator.wikimedia.org/P63970 and previous config saved to /var/cache/conftool/dbconfig/20240604-030906-marostegui.json |
[production] |
| 03:08 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2212.codfw.wmnet with reason: Maintenance |
[production] |
| 03:08 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2212.codfw.wmnet with reason: Maintenance |
[production] |
| 03:03 |
<mwpresync@deploy1002> |
Started scap: testwikis wikis to 1.43.0-wmf.8 refs T361402 |
[production] |
| 00:21 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:21 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:21 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P63969 and previous config saved to /var/cache/conftool/dbconfig/20240604-002119-ladsgroup.json |
[production] |
| 00:06 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P63968 and previous config saved to /var/cache/conftool/dbconfig/20240604-000612-ladsgroup.json |
[production] |