|
2023-05-03
ยง
|
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 100%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47320 and previous config saved to /var/cache/conftool/dbconfig/20230503-092847-root.json |
[production] |
| 09:28 |
<volans@cumin1001> |
START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.6.2 - volans@cumin1001 |
[production] |
| 09:26 |
<cgoubert@cumin1001> |
START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw |
[production] |
| 09:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2124', diff saved to https://phabricator.wikimedia.org/P47319 and previous config saved to /var/cache/conftool/dbconfig/20230503-092513-root.json |
[production] |
| 09:25 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2124.codfw.wmnet with reason: Migrating to 10.6 and rebooting |
[production] |
| 09:25 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet |
[production] |
| 09:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2124.codfw.wmnet with reason: Migrating to 10.6 and rebooting |
[production] |
| 09:24 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org |
[production] |
| 09:21 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet |
[production] |
| 09:20 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:914373|[Growth] Add GEMentorDashboardEnabledModules (T334630)]] (duration: 06m 56s) |
[production] |
| 09:20 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet |
[production] |
| 09:17 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org |
[production] |
| 09:17 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet |
[production] |
| 09:13 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:914373|[Growth] Add GEMentorDashboardEnabledModules (T334630)]] |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 75%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47318 and previous config saved to /var/cache/conftool/dbconfig/20230503-091352-root.json |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 75%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47317 and previous config saved to /var/cache/conftool/dbconfig/20230503-091342-root.json |
[production] |
| 09:12 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2001.codfw.wmnet |
[production] |
| 09:11 |
<urbanecm@deploy1002> |
sync-world aborted: Backport for [[gerrit:914426|Personalized praise: Let mentors to skip suggestions (T334300)]] (duration: 00m 06s) |
[production] |
| 09:11 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:914426|Personalized praise: Let mentors to skip suggestions (T334300)]] |
[production] |
| 09:08 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host acmechief2001.codfw.wmnet |
[production] |
| 09:05 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1001.eqiad.wmnet |
[production] |
| 09:02 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host acmechief1001.eqiad.wmnet |
[production] |
| 09:01 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host acmechief1001.eqiad.wmnet |
[production] |
| 09:01 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host acmechief1001.eqiad.wmnet |
[production] |
| 09:00 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:914426|Personalized praise: Let mentors to skip suggestions (T334300)]] (duration: 27m 39s) |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 50%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47316 and previous config saved to /var/cache/conftool/dbconfig/20230503-085847-root.json |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 50%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47315 and previous config saved to /var/cache/conftool/dbconfig/20230503-085837-root.json |
[production] |
| 08:44 |
<jelto@cumin1001> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 25%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47314 and previous config saved to /var/cache/conftool/dbconfig/20230503-084342-root.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 25%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47313 and previous config saved to /var/cache/conftool/dbconfig/20230503-084332-root.json |
[production] |
| 08:39 |
<marostegui> |
dbmaint deploy schema change on eqiad s3 with replication T335834 |
[production] |
| 08:39 |
<jelto@cumin1001> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Install software version upgrade |
[production] |
| 08:32 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:914426|Personalized praise: Let mentors to skip suggestions (T334300)]] |
[production] |
| 08:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 10%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47312 and previous config saved to /var/cache/conftool/dbconfig/20230503-082837-root.json |
[production] |
| 08:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 10%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47311 and previous config saved to /var/cache/conftool/dbconfig/20230503-082827-root.json |
[production] |
| 08:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 5%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47310 and previous config saved to /var/cache/conftool/dbconfig/20230503-081332-root.json |
[production] |
| 08:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 5%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47309 and previous config saved to /var/cache/conftool/dbconfig/20230503-081323-root.json |
[production] |
| 08:04 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org |
[production] |
| 07:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 4%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47308 and previous config saved to /var/cache/conftool/dbconfig/20230503-075828-root.json |
[production] |
| 07:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 4%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47307 and previous config saved to /var/cache/conftool/dbconfig/20230503-075818-root.json |
[production] |
| 07:57 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org |
[production] |
| 07:48 |
<jelto@cumin1001> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Install software version upgrade |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 3%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47306 and previous config saved to /var/cache/conftool/dbconfig/20230503-074323-root.json |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 3%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47305 and previous config saved to /var/cache/conftool/dbconfig/20230503-074313-root.json |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1110 T335011', diff saved to https://phabricator.wikimedia.org/P47304 and previous config saved to /var/cache/conftool/dbconfig/20230503-073602-root.json |
[production] |
| 07:28 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:914429|Remove duplicated diff-mode selector in save dialog (T324759)]] (duration: 10m 14s) |
[production] |
| 07:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 2%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47303 and previous config saved to /var/cache/conftool/dbconfig/20230503-072818-root.json |
[production] |
| 07:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 2%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47302 and previous config saved to /var/cache/conftool/dbconfig/20230503-072808-root.json |
[production] |
| 07:26 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . |
[production] |
| 07:20 |
<taavi@deploy1002> |
taavi and samwilson: Backport for [[gerrit:914429|Remove duplicated diff-mode selector in save dialog (T324759)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |