|
2021-04-27
ยง
|
| 13:46 |
<akosiaris@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 13:45 |
<akosiaris@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 13:45 |
<akosiaris> |
switchover api-gateway, changeprop, cpjobqueue to use the new redis cluster servers (rdb2007-rdb2010) |
[production] |
| 13:45 |
<akosiaris@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 13:45 |
<akosiaris@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 13:44 |
<akosiaris@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'production' . |
[production] |
| 13:44 |
<akosiaris@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'staging' . |
[production] |
| 13:34 |
<akosiaris@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 13:34 |
<akosiaris@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 13:33 |
<akosiaris@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 13:33 |
<akosiaris@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 13:30 |
<hashar> |
Upgrading CI Jenkins from 2.263.3 to 2.277.2 |
[production] |
| 13:23 |
<jayme@cumin1001> |
START - Cookbook sre.kafka.roll-restart-brokers |
[production] |
| 13:21 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[1020-1026].eqiad.wmnet |
[production] |
| 13:19 |
<jayme@cumin1001> |
START - Cookbook sre.kafka.roll-restart-brokers |
[production] |
| 13:13 |
<liw@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.3 |
[production] |
| 13:08 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.3/extensions/GrowthExperiments/includes/Config/WikiPageConfigValidation.php: fe2a0420fd884df7046c0c283bcb2e961e74e8e9: WikiPageConfigValidation: Mentor lists and help desk can be null (T281229) (duration: 01m 06s) |
[production] |
| 13:07 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf[2004-2006].codfw.wmnet with reason: for zookeeper migration |
[production] |
| 13:07 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on conf[2004-2006].codfw.wmnet with reason: for zookeeper migration |
[production] |
| 13:06 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts ms-be[1020-1026].eqiad.wmnet |
[production] |
| 13:05 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be1019.eqiad.wmnet |
[production] |
| 12:55 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts ms-be1019.eqiad.wmnet |
[production] |
| 12:46 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:682815|Revert "URGENT: Disable GlobalUsage" (T281242)]] (duration: 01m 08s) |
[production] |
| 12:44 |
<hashar> |
Restarted CI Jenkins for plugins upgrade |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15592 and previous config saved to /var/cache/conftool/dbconfig/20210427-122619-root.json |
[production] |
| 12:20 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.1/extensions/GlobalUsage: Backport: [[gerrit:682814|Avoid reading primary unless absolutely necessary (T281238)]] (duration: 01m 09s) |
[production] |
| 12:12 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.3/extensions/GlobalUsage: Backport: [[gerrit:682813|Avoid reading primary unless absolutely necessary (T281238)]] (duration: 01m 09s) |
[production] |
| 12:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15591 and previous config saved to /var/cache/conftool/dbconfig/20210427-121115-root.json |
[production] |
| 12:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on labstore1007.wikimedia.org with reason: T281045 |
[production] |
| 12:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on labstore1007.wikimedia.org with reason: T281045 |
[production] |
| 11:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 50%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15590 and previous config saved to /var/cache/conftool/dbconfig/20210427-115612-root.json |
[production] |
| 11:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: Repool db1157', diff saved to https://phabricator.wikimedia.org/P15589 and previous config saved to /var/cache/conftool/dbconfig/20210427-114108-root.json |
[production] |
| 11:36 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) |
[production] |
| 11:30 |
<elukey@cumin1001> |
START - Cookbook sre.kafka.roll-restart-mirror-maker |
[production] |
| 11:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove RW from commonswiki', diff saved to https://phabricator.wikimedia.org/P15588 and previous config saved to /var/cache/conftool/dbconfig/20210427-111016-marostegui.json |
[production] |
| 11:09 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Disable GlobalUsage (duration: 01m 08s) |
[production] |
| 10:40 |
<volans@cumin1001> |
dbctl commit (dc=all): 'S4 RO, outage', diff saved to https://phabricator.wikimedia.org/P15585 and previous config saved to /var/cache/conftool/dbconfig/20210427-104057-volans.json |
[production] |
| 10:18 |
<godog> |
swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836 |
[production] |
| 10:06 |
<XioNoX> |
standardize management routers ACLs with Capirca - mr1-eqiad (last one) |
[production] |
| 10:01 |
<ayounsi@deploy1002> |
Finished deploy [homer/deploy@759f82c]: Homer release v0.2.7 (duration: 02m 16s) |
[production] |
| 09:59 |
<ayounsi@deploy1002> |
Started deploy [homer/deploy@759f82c]: Homer release v0.2.7 |
[production] |
| 09:56 |
<ayounsi@deploy1002> |
Finished deploy [homer/deploy@759f82c]: Homer release v0.2.7 (duration: 00m 22s) |
[production] |
| 09:56 |
<ayounsi@deploy1002> |
Started deploy [homer/deploy@759f82c]: Homer release v0.2.7 |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1157 for schema change', diff saved to https://phabricator.wikimedia.org/P15584 and previous config saved to /var/cache/conftool/dbconfig/20210427-093536-marostegui.json |
[production] |
| 09:35 |
<XioNoX> |
standardize management routers ACLs with Capirca - mr1-eqsin |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Repool db1175', diff saved to https://phabricator.wikimedia.org/P15583 and previous config saved to /var/cache/conftool/dbconfig/20210427-093501-root.json |
[production] |
| 09:34 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1012.eqiad.wmnet |
[production] |
| 09:34 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1011.eqiad.wmnet |
[production] |
| 09:33 |
<moritzm> |
rolling restart of elastic in relforge* to pick up Java updates |
[production] |
| 09:32 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2010.codfw.wmnet |
[production] |