1-50 of 10000 results (25ms)
2026-05-26 ยง
16:45 <jclark@cumin1003> START - Cookbook sre.hosts.provision for host wdqs1037.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:45 <jclark@cumin1003> START - Cookbook sre.hosts.provision for host wdqs1036.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:45 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
16:45 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
16:44 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
16:44 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
16:44 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db2209 (T426633)', diff saved to https://phabricator.wikimedia.org/P93053 and previous config saved to /var/cache/conftool/dbconfig/20260526-164421-fceratto.json [production]
16:44 <jclark@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:44 <jclark@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" [production]
16:44 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance [production]
16:44 <jclark@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1002 to eqiad - jclark@cumin1003" [production]
16:43 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2194 (T426633)', diff saved to https://phabricator.wikimedia.org/P93052 and previous config saved to /var/cache/conftool/dbconfig/20260526-164352-fceratto.json [production]
16:42 <brett@cumin2002> cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet [production]
16:41 <brett@cumin2002> cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet [production]
16:40 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
16:40 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
16:40 <jclark@cumin1003> START - Cookbook sre.dns.netbox [production]
16:40 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
16:40 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
16:40 <brett> reboot lvs 101[345].eqiad.wmnet [production]
16:39 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
16:39 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
16:39 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
16:39 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
16:37 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
16:37 <jayme@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
16:37 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
16:37 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
16:37 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
16:37 <jayme@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
16:37 <jayme@deploy1003> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
16:36 <jayme@deploy1003> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
16:36 <arlolra@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
16:36 <arlolra@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
16:36 <arlolra@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
16:36 <arlolra@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
16:35 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
16:34 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
16:34 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
16:34 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
16:34 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
16:34 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
16:33 <brett@cumin2002> START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw and A:cp [production]
16:33 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P93051 and previous config saved to /var/cache/conftool/dbconfig/20260526-163344-fceratto.json [production]
16:33 <brett@cumin2002> START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw and A:cp [production]
16:31 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
16:31 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
16:30 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.88}' [admin]
16:30 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service} AND NOT P{F:kernelversion = 6.12.88}' [admin]
16:30 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]