|
2025-06-17
ยง
|
| 16:52 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P78224 and previous config saved to /var/cache/conftool/dbconfig/20250617-165253-ladsgroup.json |
[production] |
| 16:52 |
<ejegg> |
civicrm upgraded from ec2cd980 to d592e64c |
[production] |
| 16:47 |
<jiji@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM wikikube-worker-exp2001.codfw.wmnet - jiji@cumin1002" |
[production] |
| 16:47 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet |
[production] |
| 16:37 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214 (T382778)', diff saved to https://phabricator.wikimedia.org/P78223 and previous config saved to /var/cache/conftool/dbconfig/20250617-163746-ladsgroup.json |
[production] |
| 16:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1214 (T382778)', diff saved to https://phabricator.wikimedia.org/P78222 and previous config saved to /var/cache/conftool/dbconfig/20250617-163434-ladsgroup.json |
[production] |
| 16:34 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1214.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209 (T382778)', diff saved to https://phabricator.wikimedia.org/P78221 and previous config saved to /var/cache/conftool/dbconfig/20250617-163412-ladsgroup.json |
[production] |
| 16:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T396130)', diff saved to https://phabricator.wikimedia.org/P78220 and previous config saved to /var/cache/conftool/dbconfig/20250617-163231-marostegui.json |
[production] |
| 16:25 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1016.eqiad.wmnet with reason: host reimage |
[production] |
| 16:25 |
<cdobbins@cumin2002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp30[66-81].esams.wmnet} and A:cp - Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0 (T396581) |
[production] |
| 16:24 |
<ChrisDobbins901_> |
cdobbins@cumin2002:~$ sudo -i cookbook sre.cdn.roll-upgrade-varnish --query 'P{cp30[66-81].esams.wmnet}' --reason 'Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0' --task-id T396581 |
[production] |
| 16:24 |
<ChrisDobbins901_> |
cdobbins@cumin2002:~$ sudo -i cookbook --dry-run sre.cdn.roll-upgrade-varnish --query 'P{cp30[66-81].esams.wmnet}' --reason 'Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0' --task-id T396581 |
[production] |
| 16:22 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for ms-backup2001.codfw.wmnet: Renew puppet certificate - root@cumin1002 |
[production] |
| 16:22 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on lvs1016.eqiad.wmnet with reason: host reimage |
[production] |
| 16:19 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P78218 and previous config saved to /var/cache/conftool/dbconfig/20250617-161904-ladsgroup.json |
[production] |
| 16:18 |
<sukhe@cumin1002> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:ulsfo and A:cp - 9.2.10 upgrade (T390912) |
[production] |
| 16:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P78217 and previous config saved to /var/cache/conftool/dbconfig/20250617-161723-marostegui.json |
[production] |
| 16:08 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs1016.eqiad.wmnet with OS bullseye |
[production] |
| 16:05 |
<swfrench@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 16:05 |
<swfrench@deploy1003> |
helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 16:03 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P78216 and previous config saved to /var/cache/conftool/dbconfig/20250617-160357-ladsgroup.json |
[production] |
| 16:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P78215 and previous config saved to /var/cache/conftool/dbconfig/20250617-160216-marostegui.json |
[production] |
| 16:01 |
<brett@cumin2002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-drmrs and A:cp - Fix VSLbs() assert error and upgrade libvmod-wmfuniq to 0.2.0 (T396581) |
[production] |
| 15:54 |
<cdobbins@dns1004> |
END - running authdns-update |
[production] |
| 15:53 |
<cdobbins@dns1004> |
START - running authdns-update |
[production] |
| 15:48 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1209 (T382778)', diff saved to https://phabricator.wikimedia.org/P78214 and previous config saved to /var/cache/conftool/dbconfig/20250617-154850-ladsgroup.json |
[production] |
| 15:47 |
<swfrench@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 15:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T396130)', diff saved to https://phabricator.wikimedia.org/P78213 and previous config saved to /var/cache/conftool/dbconfig/20250617-154709-marostegui.json |
[production] |
| 15:47 |
<swfrench@deploy1003> |
helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 15:45 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1209 (T382778)', diff saved to https://phabricator.wikimedia.org/P78212 and previous config saved to /var/cache/conftool/dbconfig/20250617-154542-ladsgroup.json |
[production] |
| 15:45 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1209.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:45 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203 (T382778)', diff saved to https://phabricator.wikimedia.org/P78211 and previous config saved to /var/cache/conftool/dbconfig/20250617-154520-ladsgroup.json |
[production] |
| 15:33 |
<effie> |
stopping puppet on A:wikikube-worker and A:eqiad |
[production] |
| 15:30 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P78210 and previous config saved to /var/cache/conftool/dbconfig/20250617-153013-ladsgroup.json |
[production] |
| 15:29 |
<dancy@deploy1003> |
Finished scap sync-world: Testing T396166 (duration: 03m 46s) |
[production] |
| 15:28 |
<swfrench@deploy1003> |
helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 15:28 |
<swfrench@deploy1003> |
helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 15:28 |
<jynus@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-backup2001.codfw.wmnet with reason: Maintenance and reboot |
[production] |
| 15:26 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1047.eqiad.wmnet to cluster eqiad and group C |
[production] |
| 15:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2216 (T396130)', diff saved to https://phabricator.wikimedia.org/P78209 and previous config saved to /var/cache/conftool/dbconfig/20250617-152555-marostegui.json |
[production] |
| 15:25 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2216.codfw.wmnet with reason: Maintenance |
[production] |
| 15:25 |
<dancy@deploy1003> |
Started scap sync-world: Testing T396166 |
[production] |
| 15:24 |
<dancy@deploy1003> |
Installation of scap version "4.177.0" completed for 2 hosts |
[production] |
| 15:24 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum7004.magru.wmnet |
[production] |
| 15:24 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host durum7004.magru.wmnet with OS bookworm |
[production] |
| 15:23 |
<swfrench@deploy1003> |
helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 15:23 |
<swfrench@deploy1003> |
helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply |
[production] |
| 15:22 |
<dancy@deploy1003> |
Installing scap version "4.177.0" for 2 host(s) |
[production] |
| 15:15 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P78208 and previous config saved to /var/cache/conftool/dbconfig/20250617-151505-ladsgroup.json |
[production] |