|
2024-03-13
ยง
|
| 15:24 |
<marostegui@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - marostegui@cumin1002" |
[production] |
| 15:22 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Continuing with sync |
[production] |
| 15:21 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 15:19 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] |
[production] |
| 15:04 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage |
[production] |
| 15:01 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage |
[production] |
| 14:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Cloning db1246 |
[production] |
| 14:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Cloning db1246 |
[production] |
| 14:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1182 to clone db1246', diff saved to https://phabricator.wikimedia.org/P58778 and previous config saved to /var/cache/conftool/dbconfig/20240313-145603-marostegui.json |
[production] |
| 14:49 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:49 |
<marostegui@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:33 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:17 |
<Dreamy_Jazz> |
Starting MediaModeration scanning scripts on group2 wikis and commonswiki - https://wikitech.wikimedia.org/wiki/MediaModeration |
[production] |
| 14:16 |
<Dreamy_Jazz> |
Afternoon UTC backport window done |
[production] |
| 14:16 |
<dreamyjazz@deploy2002> |
Finished scap: Backport for [[gerrit:1010577|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]], [[gerrit:1010578|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]] (duration: 14m 56s) |
[production] |
| 14:10 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: HW issues |
[production] |
| 14:10 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: HW issues |
[production] |
| 14:05 |
<dreamyjazz@deploy2002> |
dreamyjazz: Continuing with sync |
[production] |
| 14:03 |
<dreamyjazz@deploy2002> |
dreamyjazz: Backport for [[gerrit:1010577|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]], [[gerrit:1010578|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:01 |
<dreamyjazz@deploy2002> |
Started scap: Backport for [[gerrit:1010577|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]], [[gerrit:1010578|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]] |
[production] |
| 13:57 |
<dreamyjazz@deploy2002> |
Finished scap: Backport for [[gerrit:1010645|Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false (T355213)]] (duration: 15m 33s) |
[production] |
| 13:52 |
<Dreamy_Jazz> |
Stopping MediaModeration scanning scripts for backport |
[production] |
| 13:51 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:51 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:47 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:47 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:46 |
<dreamyjazz@deploy2002> |
dreamyjazz and gergesshamon: Continuing with sync |
[production] |
| 13:46 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:46 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:46 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:46 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:45 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:45 |
<marostegui> |
Disable GTID on eqiad s1 master T358199 |
[production] |
| 13:45 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:44 |
<dreamyjazz@deploy2002> |
dreamyjazz and gergesshamon: Backport for [[gerrit:1010645|Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false (T355213)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:43 |
<marostegui> |
Disable GTID on eqiad s2 master T358199 |
[production] |
| 13:41 |
<dreamyjazz@deploy2002> |
Started scap: Backport for [[gerrit:1010645|Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false (T355213)]] |
[production] |
| 13:40 |
<marostegui> |
Disable GTID on eqiad s3 master T358199 |
[production] |
| 13:39 |
<marostegui> |
Disable GTID on eqiad s4 master T358199 |
[production] |
| 13:38 |
<dreamyjazz@deploy2002> |
Finished scap: Backport for [[gerrit:1010575|Use iterator_to_array when calling ::assertCount (T360017)]], [[gerrit:1010576|Use iterator_to_array when calling ::assertCount (T360017)]] (duration: 13m 01s) |
[production] |
| 13:37 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:37 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:36 |
<marostegui> |
Disable GTID on eqiad s5 master T358199 |
[production] |
| 13:35 |
<marostegui> |
Disable GTID on eqiad s6 master T358199 |
[production] |
| 13:31 |
<marostegui> |
Disable GTID on eqiad s7 master T358199 |
[production] |
| 13:29 |
<marostegui> |
Disable GTID on eqiad s8 master T358199 |
[production] |
| 13:27 |
<dreamyjazz@deploy2002> |
dreamyjazz: Continuing with sync |
[production] |
| 13:27 |
<dreamyjazz@deploy2002> |
dreamyjazz: Backport for [[gerrit:1010575|Use iterator_to_array when calling ::assertCount (T360017)]], [[gerrit:1010576|Use iterator_to_array when calling ::assertCount (T360017)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:25 |
<dreamyjazz@deploy2002> |
Started scap: Backport for [[gerrit:1010575|Use iterator_to_array when calling ::assertCount (T360017)]], [[gerrit:1010576|Use iterator_to_array when calling ::assertCount (T360017)]] |
[production] |
| 13:21 |
<mabualruz@deploy2002> |
Finished scap: Backport for [[gerrit:1010698|Disable night mode on history pages (T359183)]] (duration: 15m 48s) |
[production] |