|
2023-04-25
ยง
|
| 21:17 |
<mutante> |
gerrit1003 - mv /srv/gerrit/plugins/lfs /srv/gerrit/data/ T333143 T326368 |
[production] |
| 21:14 |
<mutante> |
gerrit1003 - manually replacing deploy2002 with deploy1002 in /srv/deployment/gerrit/gerrit-cache/.config to fix initial scap deployment T257317 T326368 |
[production] |
| 21:12 |
<mutante> |
once again running into T257317 when applying gerrit role to new hardware |
[production] |
| 21:06 |
<mutante> |
adding production gerrit role to new machine gerrit1003 - monitoring downtimed - but it has a service IP that is going to be added by this and cant be downtimed ? (Bug: T326368) |
[production] |
| 21:04 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on gerrit1003.wikimedia.org with reason: setup |
[production] |
| 21:04 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on gerrit1003.wikimedia.org with reason: setup |
[production] |
| 19:48 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wdqs2012.codfw.wmnet with reason: attempting WDQS stack on bullseye |
[production] |
| 19:48 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on wdqs2012.codfw.wmnet with reason: attempting WDQS stack on bullseye |
[production] |
| 19:48 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs2006.codfw.wmnet |
[production] |
| 19:48 |
<bking@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for wdqs2006.codfw.wmnet |
[production] |
| 19:46 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs2009.codfw.wmnet |
[production] |
| 19:46 |
<bking@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for wdqs2009.codfw.wmnet |
[production] |
| 19:46 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wdqs2006.codfw.wmnet with reason: attempting WDQS stack on bullseye |
[production] |
| 19:46 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on wdqs2006.codfw.wmnet with reason: attempting WDQS stack on bullseye |
[production] |
| 19:23 |
<inflatador> |
bking@cumin1001 finishing WDQS deploy...restarting `wdqs-categories` across lvs-managed hosts |
[production] |
| 18:57 |
<bking@deploy1002> |
Finished deploy [wdqs/wdqs@0e051d8]: 0.3.123 (duration: 17m 29s) |
[production] |
| 18:39 |
<bking@deploy1002> |
Started deploy [wdqs/wdqs@0e051d8]: 0.3.123 |
[production] |
| 18:18 |
<jhuneidi@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.6 refs T330212 |
[production] |
| 16:55 |
<ejegg> |
payments-wiki upgraded from 2a4c450d to f30bc859 |
[production] |
| 15:39 |
<akosiaris@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:39 |
<akosiaris@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 15:34 |
<akosiaris@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:33 |
<akosiaris@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 15:32 |
<akosiaris@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:31 |
<akosiaris@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 15:31 |
<akosiaris@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:31 |
<akosiaris@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 15:30 |
<dancy@deploy1002> |
Installation of scap version "4.50.0" completed for 1 hosts |
[production] |
| 15:30 |
<dancy@deploy1002> |
Installing scap version "4.50.0" for 1 hosts |
[production] |
| 15:28 |
<XioNoX> |
update cr2-eqsin BBIX interface |
[production] |
| 15:27 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) |
[production] |
| 15:27 |
<btullis@cumin1001> |
Added views for new wiki: azwikimedia T330442 |
[production] |
| 15:25 |
<dancy@deploy1002> |
Installing scap version "4.50.0" for 592 hosts |
[production] |
| 15:24 |
<cgoubert@cumin2002> |
END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool restbase-async in eqiad: T335015 |
[production] |
| 15:22 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on irc2002.wikimedia.org with reason: Non-functional, WIP for Bullseye update |
[production] |
| 15:22 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on irc2002.wikimedia.org with reason: Non-functional, WIP for Bullseye update |
[production] |
| 15:22 |
<claime> |
Datacenter Service Switchback concluded - T335015 |
[production] |
| 15:21 |
<cgoubert@deploy1002> |
Synchronized README: check the deployment server after switchback - T335015 (duration: 19m 55s) |
[production] |
| 15:19 |
<cgoubert@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) restbase-async.discovery.wmnet on all recursors |
[production] |
| 15:19 |
<cgoubert@cumin2002> |
START - Cookbook sre.dns.wipe-cache restbase-async.discovery.wmnet on all recursors |
[production] |
| 15:19 |
<cgoubert@cumin2002> |
START - Cookbook sre.discovery.service-route depool restbase-async in eqiad: T335015 |
[production] |
| 15:18 |
<claime> |
Restoring restbase-async to codfw only - T335015 |
[production] |
| 15:18 |
<cgoubert@deploy1002> |
Finished deploy [restbase/deploy@a08f56d]: (no justification provided) (duration: 13m 06s) |
[production] |
| 15:08 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1002.eqiad.wmnet with OS bullseye |
[production] |
| 15:05 |
<cgoubert@deploy1002> |
Started deploy [restbase/deploy@a08f56d]: (no justification provided) |
[production] |
| 15:02 |
<btullis@cumin1001> |
START - Cookbook sre.wikireplicas.add-wiki |
[production] |
| 15:02 |
<inflatador> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 15:02 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) |
[production] |
| 15:02 |
<btullis@cumin1001> |
Added views for new wiki: vewikimedia T330704 |
[production] |
| 15:01 |
<btullis@cumin1001> |
START - Cookbook sre.wikireplicas.add-wiki |
[production] |