|
2024-06-04
ยง
|
| 08:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T364069)', diff saved to https://phabricator.wikimedia.org/P63991 and previous config saved to /var/cache/conftool/dbconfig/20240604-085141-marostegui.json |
[production] |
| 08:50 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1003.wikimedia.org |
[production] |
| 08:46 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host idp-test1003.wikimedia.org |
[production] |
| 08:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1156', diff saved to https://phabricator.wikimedia.org/P63990 and previous config saved to /var/cache/conftool/dbconfig/20240604-084428-root.json |
[production] |
| 08:40 |
<kostajh> |
UTC morning deploys done |
[production] |
| 08:38 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:1038634|IPReputationHooks: Bump schema version (T354597)]] (duration: 15m 45s) |
[production] |
| 08:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P63989 and previous config saved to /var/cache/conftool/dbconfig/20240604-083633-marostegui.json |
[production] |
| 08:19 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:1038633|IPReputationHooks: Bump schema version (T354597)]] (duration: 14m 08s) |
[production] |
| 08:10 |
<kharlan@deploy1002> |
kharlan: Continuing with sync |
[production] |
| 08:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P63986 and previous config saved to /var/cache/conftool/dbconfig/20240604-080846-marostegui.json |
[production] |
| 08:08 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:1038633|IPReputationHooks: Bump schema version (T354597)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T364069)', diff saved to https://phabricator.wikimedia.org/P63985 and previous config saved to /var/cache/conftool/dbconfig/20240604-080617-marostegui.json |
[production] |
| 08:06 |
<jiji@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage |
[production] |
| 08:05 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:1038633|IPReputationHooks: Bump schema version (T354597)]] |
[production] |
| 08:02 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage |
[production] |
| 08:01 |
<jiji@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf2002.codfw.wmnet with reason: host reimage |
[production] |
| 07:57 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc-wf1002.eqiad.wmnet with reason: host reimage |
[production] |
| 07:56 |
<hashar> |
Restarting Gerrit for Java 17 upgrade # T364342 |
[production] |
| 07:56 |
<hashar@deploy1002> |
Finished deploy [gerrit/gerrit@6ba3f2e]: gerrit1003: switch to Java 17 version of plugins after having switched Java to 17- T364342 (duration: 00m 03s) |
[production] |
| 07:55 |
<hashar@deploy1002> |
Started deploy [gerrit/gerrit@6ba3f2e]: gerrit1003: switch to Java 17 version of plugins after having switched Java to 17- T364342 |
[production] |
| 07:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P63984 and previous config saved to /var/cache/conftool/dbconfig/20240604-075338-marostegui.json |
[production] |
| 07:47 |
<hashar@deploy1002> |
Finished deploy [gerrit/gerrit@6ba3f2e]: gerrit2002: switch to Java 17 version of plugins after having switched Java to 17- T364342 (duration: 00m 05s) |
[production] |
| 07:46 |
<hashar@deploy1002> |
Started deploy [gerrit/gerrit@6ba3f2e]: gerrit2002: switch to Java 17 version of plugins after having switched Java to 17- T364342 |
[production] |
| 07:42 |
<jiji@cumin2002> |
START - Cookbook sre.hosts.reimage for host mc-wf2002.codfw.wmnet with OS bookworm |
[production] |
| 07:42 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.reimage for host mc-wf1002.eqiad.wmnet with OS bookworm |
[production] |
| 07:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T364299)', diff saved to https://phabricator.wikimedia.org/P63983 and previous config saved to /var/cache/conftool/dbconfig/20240604-073830-marostegui.json |
[production] |
| 07:27 |
<marostegui> |
dbmaint eqiad s1 deploy schema change on db1184 T356166 |
[production] |
| 07:15 |
<moritzm> |
installing intel-microcode updates on bullseye |
[production] |
| 07:10 |
<marostegui> |
dbmaint eqiad s1 deploy schema change on db1184 T355609 |
[production] |
| 07:06 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:06 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:05 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS bookworm |
[production] |
| 06:43 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage |
[production] |
| 06:40 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage |
[production] |
| 06:26 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS bookworm |
[production] |
| 06:26 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1184.eqiad.wmnet with reason: reimage |
[production] |
| 06:26 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on db1184.eqiad.wmnet with reason: reimage |
[production] |
| 06:14 |
<marostegui> |
Rename table flaggedpage_pending on db1185 (s5 eqiad dbmaint) - T365568 |
[production] |
| 06:09 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): ' fix api db1163 vs db1184 T366259', diff saved to https://phabricator.wikimedia.org/P63982 and previous config saved to /var/cache/conftool/dbconfig/20240604-060925-arnaudb.json |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'API db1163 T366259', diff saved to https://phabricator.wikimedia.org/P63981 and previous config saved to /var/cache/conftool/dbconfig/20240604-060747-arnaudb.json |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depool db1184 T366259', diff saved to https://phabricator.wikimedia.org/P63980 and previous config saved to /var/cache/conftool/dbconfig/20240604-060703-arnaudb.json |
[production] |
| 06:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Promote db1163 to s1 primary and set section read-write T366259', diff saved to https://phabricator.wikimedia.org/P63979 and previous config saved to /var/cache/conftool/dbconfig/20240604-060324-arnaudb.json |
[production] |
| 06:02 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T366259', diff saved to https://phabricator.wikimedia.org/P63978 and previous config saved to /var/cache/conftool/dbconfig/20240604-060208-arnaudb.json |
[production] |
| 06:01 |
<arnaudb> |
Starting s1 eqiad failover from db1184 to db1163 - T366259 |
[production] |
| 05:28 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Set db1163 with weight 0 T366259', diff saved to https://phabricator.wikimedia.org/P63977 and previous config saved to /var/cache/conftool/dbconfig/20240604-052803-arnaudb.json |
[production] |
| 05:27 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 35 hosts with reason: Primary switchover s1 T366259 |
[production] |
| 05:27 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 35 hosts with reason: Primary switchover s1 T366259 |
[production] |
| 04:20 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1246 (T352010)', diff saved to https://phabricator.wikimedia.org/P63976 and previous config saved to /var/cache/conftool/dbconfig/20240604-042011-ladsgroup.json |
[production] |
| 04:20 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:19 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1246.eqiad.wmnet with reason: Maintenance |
[production] |