|
2024-06-04
ยง
|
| 21:28 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/data/discovery/wikidata/munged_n3_dump/wikidata/full/20240527/ using stat1009.eqiad.wmnet) |
[production] |
| 21:24 |
<urbanecm@deploy1002> |
toyofuku and urbanecm: Continuing with sync |
[production] |
| 21:21 |
<urbanecm@deploy1002> |
toyofuku and urbanecm: Backport for [[gerrit:1038444|Disable font size options on specified pages for most wikis (T366334)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:18 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:1038444|Disable font size options on specified pages for most wikis (T366334)]] |
[production] |
| 21:10 |
<tgr@deploy1002> |
Finished scap: Backport for [[gerrit:1037929|multiversion: Support beta for upload hostname check]], [[gerrit:1037930|multiversion: Add tests for MWMultiVersion::getMediaWiki()]] (duration: 16m 33s) |
[production] |
| 21:07 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-ctrl1001.eqiad.wmnet with OS bullseye |
[production] |
| 21:06 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wikikube-ctrl1001'] |
[production] |
| 21:01 |
<tgr@deploy1002> |
tgr: Continuing with sync |
[production] |
| 20:58 |
<tgr@deploy1002> |
tgr: Backport for [[gerrit:1037929|multiversion: Support beta for upload hostname check]], [[gerrit:1037930|multiversion: Add tests for MWMultiVersion::getMediaWiki()]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:56 |
<kamila@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wikikube-ctrl1001'] |
[production] |
| 20:53 |
<tgr@deploy1002> |
Started scap: Backport for [[gerrit:1037929|multiversion: Support beta for upload hostname check]], [[gerrit:1037930|multiversion: Add tests for MWMultiVersion::getMediaWiki()]] |
[production] |
| 20:52 |
<kamila@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl1001.eqiad.wmnet with OS bullseye |
[production] |
| 20:47 |
<tgr@deploy1002> |
Finished scap: Backport for [[gerrit:1035749|beta: Introduce new test2wiki on test2.wikipedia.beta.wmcloud.org (T355281)]] (duration: 13m 12s) |
[production] |
| 20:39 |
<tgr@deploy1002> |
tgr and pmiazga: Continuing with sync |
[production] |
| 20:37 |
<tgr@deploy1002> |
tgr and pmiazga: Backport for [[gerrit:1035749|beta: Introduce new test2wiki on test2.wikipedia.beta.wmcloud.org (T355281)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:34 |
<tgr@deploy1002> |
Started scap: Backport for [[gerrit:1035749|beta: Introduce new test2wiki on test2.wikipedia.beta.wmcloud.org (T355281)]] |
[production] |
| 20:28 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:1037945|[pawiki] Enable wgMinervaEnableSiteNotice (T366434)]] (duration: 13m 24s) |
[production] |
| 20:27 |
<jhathaway> |
vacuuming pcc db |
[production] |
| 20:26 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 20:25 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 20:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137 (T364299)', diff saved to https://phabricator.wikimedia.org/P64039 and previous config saved to /var/cache/conftool/dbconfig/20240604-202554-marostegui.json |
[production] |
| 20:22 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts mw1358.eqiad.wmnet |
[production] |
| 20:22 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts mw1358.eqiad.wmnet |
[production] |
| 20:21 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts mw1358.eqiad.wmnet |
[production] |
| 20:21 |
<jclark@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts mw1358.eqiad.wmnet |
[production] |
| 20:19 |
<ladsgroup@deploy1002> |
pppery and ladsgroup: Continuing with sync |
[production] |
| 20:17 |
<ladsgroup@deploy1002> |
pppery and ladsgroup: Backport for [[gerrit:1037945|[pawiki] Enable wgMinervaEnableSiteNotice (T366434)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:15 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:1037945|[pawiki] Enable wgMinervaEnableSiteNotice (T366434)]] |
[production] |
| 20:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P64038 and previous config saved to /var/cache/conftool/dbconfig/20240604-201047-marostegui.json |
[production] |
| 20:00 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-ctrl1001.eqiad.wmnet with OS bullseye |
[production] |
| 19:59 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wikikube-ctrl1001'] |
[production] |
| 19:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P64037 and previous config saved to /var/cache/conftool/dbconfig/20240604-195539-marostegui.json |
[production] |
| 19:49 |
<ecarg@deploy1002> |
helmfile [staging] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 19:49 |
<ecarg@deploy1002> |
helmfile [staging] START helmfile.d/services/wikifunctions: apply |
[production] |
| 19:47 |
<kamila@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wikikube-ctrl1001'] |
[production] |
| 19:44 |
<kamila@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl1001.eqiad.wmnet with OS bullseye |
[production] |
| 19:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2137 (T364299)', diff saved to https://phabricator.wikimedia.org/P64036 and previous config saved to /var/cache/conftool/dbconfig/20240604-194031-marostegui.json |
[production] |
| 19:38 |
<mutante> |
https://gerrit-replica.wikimedia.org - short downtime for maintenance |
[production] |
| 19:38 |
<dzahn@cumin1002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:10:00 on gerrit-replica.wikimedia.org with reason: reboot T366555 |
[production] |
| 19:38 |
<dzahn@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:10:00 on gerrit-replica.wikimedia.org with reason: reboot T366555 |
[production] |
| 19:37 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-ctrl1001.eqiad.wmnet with OS bullseye |
[production] |
| 19:37 |
<dzahn@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2002.wikimedia.org with reason: reboot T366555 |
[production] |
| 19:37 |
<kamila@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-ctrl1001.eqiad.wmnet with OS bullseye |
[production] |
| 19:37 |
<dzahn@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:10:00 on gerrit2002.wikimedia.org with reason: reboot T366555 |
[production] |
| 19:36 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-ctrl1001.eqiad.wmnet with OS bullseye |
[production] |
| 19:33 |
<dzahn@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on contint2002.wikimedia.org with reason: reboot T366555 |
[production] |
| 19:32 |
<dzahn@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:10:00 on contint2002.wikimedia.org with reason: reboot T366555 |
[production] |
| 19:16 |
<mutante> |
releases.wikimedia.org - short downtime for maintenance |
[production] |
| 19:14 |
<dzahn@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on releases1003.eqiad.wmnet with reason: reboot T366555 |
[production] |
| 19:13 |
<dzahn@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:10:00 on releases1003.eqiad.wmnet with reason: reboot T366555 |
[production] |