|
2023-12-11
ยง
|
| 10:38 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/recommendation-api: sync |
[production] |
| 10:38 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/recommendation-api: sync |
[production] |
| 10:37 |
<claime> |
Repooling dse-k8s-worker nodes - sudo confctl select "service=kubesvc,cluster=dse-k8s" set/pooled=yes - T352639 |
[production] |
| 10:03 |
<jayme> |
removed cergen certs of all k8s servies from private puppet in commit d36a97aa23e21824f95d22264d06e2c3bf3c6ac3 - T300033 |
[production] |
| 09:57 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 38753 |
[production] |
| 09:56 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 38753 |
[production] |
| 09:55 |
<elukey@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/recommendation-api: sync |
[production] |
| 09:55 |
<elukey@deploy2002> |
helmfile [codfw] START helmfile.d/services/recommendation-api: sync |
[production] |
| 09:54 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 1547 |
[production] |
| 09:54 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 1547 |
[production] |
| 09:50 |
<elukey@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/recommendation-api: sync |
[production] |
| 09:50 |
<elukey@deploy2002> |
helmfile [eqiad] START helmfile.d/services/recommendation-api: sync |
[production] |
| 09:44 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/recommendation-api: sync |
[production] |
| 09:44 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/recommendation-api: sync |
[production] |
| 08:43 |
<kostajh> |
UTC morning deploys done |
[production] |
| 08:43 |
<kharlan@deploy2002> |
Finished scap: Backport for [[gerrit:976252|ClusterConfig: Rename `isTest()` to `isDebug()` for consistency (T347366)]], [[gerrit:981424|IPInfo: Add comment clarifying $wgIPInfoGeoIP2EnterprisePath (T304604)]] (duration: 22m 02s) |
[production] |
| 08:40 |
<dcausse> |
restarted blazegraph on wdqs1006 (BlazegraphFreeAllocatorsDecreasingRapidly) |
[production] |
| 08:36 |
<kharlan@deploy2002> |
kharlan and d3r1ck01: Continuing with sync |
[production] |
| 08:22 |
<kharlan@deploy2002> |
kharlan and d3r1ck01: Backport for [[gerrit:976252|ClusterConfig: Rename `isTest()` to `isDebug()` for consistency (T347366)]], [[gerrit:981424|IPInfo: Add comment clarifying $wgIPInfoGeoIP2EnterprisePath (T304604)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:21 |
<kharlan@deploy2002> |
Started scap: Backport for [[gerrit:976252|ClusterConfig: Rename `isTest()` to `isDebug()` for consistency (T347366)]], [[gerrit:981424|IPInfo: Add comment clarifying $wgIPInfoGeoIP2EnterprisePath (T304604)]] |
[production] |
| 08:16 |
<kharlan@deploy2002> |
Finished scap: Backport for [[gerrit:979969|MediaModeration: Set MediaModerationDeveloperMode to false]] (duration: 09m 55s) |
[production] |
| 08:15 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 08:15 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 08:09 |
<kharlan@deploy2002> |
kharlan: Continuing with sync |
[production] |
| 08:07 |
<kharlan@deploy2002> |
kharlan: Backport for [[gerrit:979969|MediaModeration: Set MediaModerationDeveloperMode to false]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:06 |
<kharlan@deploy2002> |
Started scap: Backport for [[gerrit:979969|MediaModeration: Set MediaModerationDeveloperMode to false]] |
[production] |
| 07:53 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 07:53 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 07:31 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2185.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 07:31 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2185.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 07:24 |
<arnaudb@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on db2185.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 07:24 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2185.codfw.wmnet with reason: reboot for upgrade |
[production] |
| 07:12 |
<marostegui> |
Failvoer m3-master from dbproxy1020 to dbproxy1026 T351864 |
[production] |
| 07:12 |
<marostegui> |
Failvoer m3-master from dbproxy1020 to dbproxy1026 org |
[production] |
| 06:44 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:981729|Revert "ProductionServices.php: Promote pc1014 to pc1"]] (duration: 08m 22s) |
[production] |
| 06:37 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 06:37 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:981729|Revert "ProductionServices.php: Promote pc1014 to pc1"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 06:35 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:981729|Revert "ProductionServices.php: Promote pc1014 to pc1"]] |
[production] |
| 06:35 |
<_joe_> |
update sirenbot to 0.3.7 |
[production] |
| 06:34 |
<kartik@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 06:29 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1011.eqiad.wmnet with OS bookworm |
[production] |
| 06:29 |
<kartik@deploy2002> |
helmfile [codfw] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:26 |
<kartik@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 06:19 |
<kartik@deploy2002> |
helmfile [eqiad] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:16 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 06:13 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:10 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1011.eqiad.wmnet with reason: host reimage |
[production] |
| 06:07 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on pc1011.eqiad.wmnet with reason: host reimage |
[production] |
| 06:07 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 06:07 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/machinetranslation: apply |
[production] |