|
2024-11-28
ยง
|
| 11:08 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2204.codfw.wmnet with reason: Maintenance |
[production] |
| 11:04 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling es2033 (T376905)', diff saved to https://phabricator.wikimedia.org/P71324 and previous config saved to /var/cache/conftool/dbconfig/20241128-110457-ladsgroup.json |
[production] |
| 11:04 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2033.codfw.wmnet with reason: Maintenance |
[production] |
| 11:04 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2033.codfw.wmnet with reason: Maintenance |
[production] |
| 11:03 |
<ladsgroup@cumin1002> |
START - Cookbook sre.mysql.pool db2237 gradually with 4 steps - Maint over (T379813) |
[production] |
| 10:51 |
<oblivian@cumin1002> |
END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Fix commit bug - oblivian@cumin1002" |
[production] |
| 10:51 |
<oblivian@cumin1002> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix commit bug - oblivian@cumin1002 |
[production] |
| 10:51 |
<oblivian@cumin1002> |
START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix commit bug - oblivian@cumin1002 |
[production] |
| 10:51 |
<oblivian@cumin1002> |
START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Fix commit bug - oblivian@cumin1002" |
[production] |
| 10:32 |
<isaranto@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 10:27 |
<isaranto@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 09:36 |
<jnuche@deploy2002> |
Finished deploy [releng/jenkins-deploy@34b35a5] (releasing): Update Jenkins version on releases1003.eqiad.wmnet (duration: 01m 22s) |
[production] |
| 09:35 |
<jnuche@deploy2002> |
Started deploy [releng/jenkins-deploy@34b35a5] (releasing): Update Jenkins version on releases1003.eqiad.wmnet |
[production] |
| 09:31 |
<jnuche@deploy2002> |
Finished deploy [releng/jenkins-deploy@34b35a5] (releasing): Update Jenkins version on releases2003.codfw.wmnet (duration: 01m 27s) |
[production] |
| 09:30 |
<jnuche@deploy2002> |
Started deploy [releng/jenkins-deploy@34b35a5] (releasing): Update Jenkins version on releases2003.codfw.wmnet |
[production] |
| 09:23 |
<isaranto@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 09:22 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.5 refs T375664 |
[production] |
| 09:14 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2200.codfw.wmnet with reason: Maintenance |
[production] |
| 09:13 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2200.codfw.wmnet with reason: Maintenance |
[production] |
| 09:09 |
<isaranto@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 09:06 |
<isaranto@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 09:00 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2198.codfw.wmnet with reason: Maintenance |
[production] |
| 09:00 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2198.codfw.wmnet with reason: Maintenance |
[production] |
| 09:00 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2195 (T370903)', diff saved to https://phabricator.wikimedia.org/P71319 and previous config saved to /var/cache/conftool/dbconfig/20241128-090035-ladsgroup.json |
[production] |
| 08:45 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P71318 and previous config saved to /var/cache/conftool/dbconfig/20241128-084528-ladsgroup.json |
[production] |
| 08:43 |
<isaranto@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 08:41 |
<isaranto@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 08:30 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P71317 and previous config saved to /var/cache/conftool/dbconfig/20241128-083021-ladsgroup.json |
[production] |
| 08:15 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2195 (T370903)', diff saved to https://phabricator.wikimedia.org/P71316 and previous config saved to /var/cache/conftool/dbconfig/20241128-081514-ladsgroup.json |
[production] |
| 08:02 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2195 (T370903)', diff saved to https://phabricator.wikimedia.org/P71315 and previous config saved to /var/cache/conftool/dbconfig/20241128-080244-ladsgroup.json |
[production] |
| 08:02 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2195.codfw.wmnet with reason: Maintenance |
[production] |
| 08:02 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2195.codfw.wmnet with reason: Maintenance |
[production] |
| 08:02 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T370903)', diff saved to https://phabricator.wikimedia.org/P71314 and previous config saved to /var/cache/conftool/dbconfig/20241128-080221-ladsgroup.json |
[production] |
| 07:56 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1022.eqiad.wmnet |
[production] |
| 07:47 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P71313 and previous config saved to /var/cache/conftool/dbconfig/20241128-074714-ladsgroup.json |
[production] |
| 07:32 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P71312 and previous config saved to /var/cache/conftool/dbconfig/20241128-073207-ladsgroup.json |
[production] |
| 07:23 |
<oblivian@cumin1002> |
END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "CSRF token support - oblivian@cumin1002" |
[production] |
| 07:23 |
<oblivian@cumin1002> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: CSRF token support - oblivian@cumin1002 |
[production] |
| 07:23 |
<oblivian@cumin1002> |
START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: CSRF token support - oblivian@cumin1002 |
[production] |
| 07:22 |
<oblivian@cumin1002> |
START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "CSRF token support - oblivian@cumin1002" |
[production] |
| 07:17 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T370903)', diff saved to https://phabricator.wikimedia.org/P71310 and previous config saved to /var/cache/conftool/dbconfig/20241128-071700-ladsgroup.json |
[production] |
| 07:02 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2181 (T370903)', diff saved to https://phabricator.wikimedia.org/P71309 and previous config saved to /var/cache/conftool/dbconfig/20241128-070231-ladsgroup.json |
[production] |
| 07:02 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2181.codfw.wmnet with reason: Maintenance |
[production] |
| 07:02 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2181.codfw.wmnet with reason: Maintenance |
[production] |
| 07:02 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167 (T370903)', diff saved to https://phabricator.wikimedia.org/P71308 and previous config saved to /var/cache/conftool/dbconfig/20241128-070209-ladsgroup.json |
[production] |
| 07:02 |
<kartik@deploy2002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 06:47 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P71307 and previous config saved to /var/cache/conftool/dbconfig/20241128-064702-ladsgroup.json |
[production] |
| 06:31 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P71306 and previous config saved to /var/cache/conftool/dbconfig/20241128-063155-ladsgroup.json |
[production] |
| 06:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167 (T370903)', diff saved to https://phabricator.wikimedia.org/P71305 and previous config saved to /var/cache/conftool/dbconfig/20241128-061647-ladsgroup.json |
[production] |
| 06:04 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2167 (T370903)', diff saved to https://phabricator.wikimedia.org/P71304 and previous config saved to /var/cache/conftool/dbconfig/20241128-060418-ladsgroup.json |
[production] |