|
2023-01-31
ยง
|
| 14:43 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagetcd1006.eqiad.wmnet with reason: host reimage |
[production] |
| 14:42 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp2035.codfw.wmnet with OS bullseye |
[production] |
| 14:41 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagetcd1005.eqiad.wmnet with reason: host reimage |
[production] |
| 14:41 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagetcd1004.eqiad.wmnet with reason: host reimage |
[production] |
| 14:41 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagetcd1006.eqiad.wmnet with reason: host reimage |
[production] |
| 14:34 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:885041|Disable write old for CheckUserLog reason field for testwiki (T233004)]], [[gerrit:885051|Remove redundant definition of wgCheckUserEnableSpecialInvestigate]], [[gerrit:885337|Bump parsoid parser cache writes to 25%. (T320534)]] (duration: 07m 23s) |
[production] |
| 14:33 |
<jayme@cumin1001> |
START - Cookbook sre.ganeti.reimage for host kubestagetcd1006.eqiad.wmnet with OS bullseye |
[production] |
| 14:33 |
<jayme@cumin1001> |
START - Cookbook sre.ganeti.reimage for host kubestagetcd1005.eqiad.wmnet with OS bullseye |
[production] |
| 14:32 |
<jayme@cumin1001> |
START - Cookbook sre.ganeti.reimage for host kubestagetcd1004.eqiad.wmnet with OS bullseye |
[production] |
| 14:28 |
<urbanecm@deploy1002> |
dreamyjazz and urbanecm and daniel: Backport for [[gerrit:885041|Disable write old for CheckUserLog reason field for testwiki (T233004)]], [[gerrit:885051|Remove redundant definition of wgCheckUserEnableSpecialInvestigate]], [[gerrit:885337|Bump parsoid parser cache writes to 25%. (T320534)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwde |
[production] |
| 14:26 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:885041|Disable write old for CheckUserLog reason field for testwiki (T233004)]], [[gerrit:885051|Remove redundant definition of wgCheckUserEnableSpecialInvestigate]], [[gerrit:885337|Bump parsoid parser cache writes to 25%. (T320534)]] |
[production] |
| 14:20 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:885041|Disable write old for CheckUserLog reason field for testwiki (T233004)]], [[gerrit:885051|Remove redundant definition of wgCheckUserEnableSpecialInvestigate]], [[gerrit:885337|Bump parsoid parser cache writes to 25%. (T320534)]] (duration: 16m 33s) |
[production] |
| 14:08 |
<mbsantos@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 14:07 |
<mbsantos@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mobileapps: apply |
[production] |
| 14:06 |
<mbsantos@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 14:05 |
<mbsantos@deploy1002> |
helmfile [codfw] START helmfile.d/services/mobileapps: apply |
[production] |
| 14:05 |
<mbsantos@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 14:05 |
<urbanecm@deploy1002> |
urbanecm and dreamyjazz and daniel: Backport for [[gerrit:885041|Disable write old for CheckUserLog reason field for testwiki (T233004)]], [[gerrit:885051|Remove redundant definition of wgCheckUserEnableSpecialInvestigate]], [[gerrit:885337|Bump parsoid parser cache writes to 25%. (T320534)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwde |
[production] |
| 14:05 |
<mbsantos@deploy1002> |
helmfile [staging] START helmfile.d/services/mobileapps: apply |
[production] |
| 14:04 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 6 hosts with reason: Reinitialize staging-eqiad with k8s 1.23 |
[production] |
| 14:04 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 6 hosts with reason: Reinitialize staging-eqiad with k8s 1.23 |
[production] |
| 14:03 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:885041|Disable write old for CheckUserLog reason field for testwiki (T233004)]], [[gerrit:885051|Remove redundant definition of wgCheckUserEnableSpecialInvestigate]], [[gerrit:885337|Bump parsoid parser cache writes to 25%. (T320534)]] |
[production] |
| 14:01 |
<urbanecm@deploy1002> |
Backport cancelled. |
[production] |
| 12:36 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@5c58f8f] (eqiad): Disable traffic mirroring from codfw to eqiad (duration: 00m 35s) |
[production] |
| 12:36 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@5c58f8f] (eqiad): Disable traffic mirroring from codfw to eqiad |
[production] |
| 11:51 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@42a07d3] (eqiad): Disable traffic mirroring from codfw to eqiad (duration: 00m 35s) |
[production] |
| 11:50 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@42a07d3] (eqiad): Disable traffic mirroring from codfw to eqiad |
[production] |
| 11:21 |
<moritzm> |
installing bind9 security updates (client-side tools/libs only) |
[production] |
| 10:57 |
<jayme@cumin1001> |
conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=k8s-ingress-staging |
[production] |
| 10:57 |
<jayme@cumin1001> |
conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-staging |
[production] |
| 10:18 |
<jayme> |
switching active kubernetes staging cluster from eqiad to codfw - T327664 |
[production] |
| 09:20 |
<marostegui> |
dbmaint Install MariaDB 10.6 on db2093 (db_inventory) T328408 |
[production] |
| 09:05 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . |
[production] |
| 09:00 |
<zabe@deploy1002> |
Finished scap: Backport for [[gerrit:885274|Stop writing to cuc_user and cuc_user_text in testwiki (T233004)]] (duration: 08m 11s) |
[production] |
| 09:00 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 08:54 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . |
[production] |
| 08:54 |
<elukey> |
roll restart kafka on kafka-logging1001 to pick up new pki certs |
[production] |
| 08:53 |
<zabe@deploy1002> |
zabe: Backport for [[gerrit:885274|Stop writing to cuc_user and cuc_user_text in testwiki (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 08:51 |
<zabe@deploy1002> |
Started scap: Backport for [[gerrit:885274|Stop writing to cuc_user and cuc_user_text in testwiki (T233004)]] |
[production] |
| 08:49 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 08:45 |
<elukey> |
restore previously removed password for keystore to kafka-logging clusters |
[production] |
| 08:39 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . |
[production] |
| 08:36 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . |
[production] |
| 07:56 |
<moritzm> |
installing bash bugfix updates from Bullseye point release |
[production] |
| 07:22 |
<marostegui> |
dbmaint Schema change on s3 eqiad T328373 |
[production] |
| 07:22 |
<marostegui> |
dbmaint Schema change on s1 eqiad T328373 |
[production] |
| 07:10 |
<marostegui> |
Failover m2 from db1164 to db1195 - T328253 |
[production] |
| 07:06 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2133,2160].codfw.wmnet,db[1117,1164,1195].eqiad.wmnet with reason: Primary switchover m2 T328253 |
[production] |
| 07:06 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db[2133,2160].codfw.wmnet,db[1117,1164,1195].eqiad.wmnet with reason: Primary switchover m2 T328253 |
[production] |
| 07:03 |
<marostegui> |
dbmaint Schema change on s5 eqiad T328373 |
[production] |