|
2024-03-13
§
|
| 17:00 |
<elukey@deploy2002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . |
[production] |
| 16:52 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1182.eqiad.wmnet onto db1246.eqiad.wmnet |
[production] |
| 16:48 |
<root@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None |
[production] |
| 16:48 |
<root@cumin1002> |
START - Cookbook sre.discovery.datacenter status all services in all: None - None |
[production] |
| 16:33 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase1021.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 16:33 |
<eevans@cumin1002> |
START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase1021.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 16:32 |
<root@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None |
[production] |
| 16:32 |
<root@cumin1002> |
START - Cookbook sre.discovery.datacenter status all services in all: None - None |
[production] |
| 16:07 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 16:07 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 16:01 |
<hashar@deploy2002> |
Synchronized php: group1 wikis to 1.42.0-wmf.22 refs T354440 (duration: 11m 46s) |
[production] |
| 15:49 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.22 refs T354440 |
[production] |
| 15:32 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] (duration: 13m 27s) |
[production] |
| 15:27 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.clone of db1182.eqiad.wmnet onto db1246.eqiad.wmnet |
[production] |
| 15:25 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 15:25 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - marostegui@cumin1002" |
[production] |
| 15:24 |
<marostegui@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - marostegui@cumin1002" |
[production] |
| 15:22 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Continuing with sync |
[production] |
| 15:21 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 15:19 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] |
[production] |
| 15:04 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage |
[production] |
| 15:01 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage |
[production] |
| 14:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Cloning db1246 |
[production] |
| 14:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Cloning db1246 |
[production] |
| 14:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1182 to clone db1246', diff saved to https://phabricator.wikimedia.org/P58778 and previous config saved to /var/cache/conftool/dbconfig/20240313-145603-marostegui.json |
[production] |
| 14:49 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:49 |
<marostegui@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:33 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:17 |
<Dreamy_Jazz> |
Starting MediaModeration scanning scripts on group2 wikis and commonswiki - https://wikitech.wikimedia.org/wiki/MediaModeration |
[production] |
| 14:16 |
<Dreamy_Jazz> |
Afternoon UTC backport window done |
[production] |
| 14:16 |
<dreamyjazz@deploy2002> |
Finished scap: Backport for [[gerrit:1010577|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]], [[gerrit:1010578|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]] (duration: 14m 56s) |
[production] |
| 14:10 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: HW issues |
[production] |
| 14:10 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: HW issues |
[production] |
| 14:05 |
<dreamyjazz@deploy2002> |
dreamyjazz: Continuing with sync |
[production] |
| 14:03 |
<dreamyjazz@deploy2002> |
dreamyjazz: Backport for [[gerrit:1010577|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]], [[gerrit:1010578|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:01 |
<dreamyjazz@deploy2002> |
Started scap: Backport for [[gerrit:1010577|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]], [[gerrit:1010578|Use wgCanonicalServer instead of wgSitename in intro text of email (T359979)]] |
[production] |
| 13:57 |
<dreamyjazz@deploy2002> |
Finished scap: Backport for [[gerrit:1010645|Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false (T355213)]] (duration: 15m 33s) |
[production] |
| 13:52 |
<Dreamy_Jazz> |
Stopping MediaModeration scanning scripts for backport |
[production] |
| 13:51 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:51 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:47 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:47 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:46 |
<dreamyjazz@deploy2002> |
dreamyjazz and gergesshamon: Continuing with sync |
[production] |
| 13:46 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:46 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:46 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:46 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:45 |
<jclark@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |
| 13:45 |
<marostegui> |
Disable GTID on eqiad s1 master T358199 |
[production] |
| 13:45 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet |
[production] |