1001-1050 of 10000 results (36ms)
2025-05-26 ยง
16:14 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/changeprop: apply [production]
16:14 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/changeprop: apply [production]
16:13 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/changeprop: apply [production]
16:07 <cgoubert@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [production]
16:07 <cgoubert@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-cron: apply [production]
16:04 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P76429 and previous config saved to /var/cache/conftool/dbconfig/20250526-160447-fceratto.json [production]
15:49 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76428 and previous config saved to /var/cache/conftool/dbconfig/20250526-154939-fceratto.json [production]
15:49 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
15:48 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
15:48 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
15:47 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
15:47 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
15:47 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
15:46 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
15:46 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
15:46 <volans@cumin1003> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cirrussearch2111.codfw.wmnet [production]
15:45 <volans@cumin1003> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cirrussearch2111.codfw.wmnet [production]
15:45 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
15:44 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
15:43 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db2157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76427 and previous config saved to /var/cache/conftool/dbconfig/20250526-154304-fceratto.json [production]
15:42 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
15:32 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
15:31 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [production]
15:29 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
15:29 <volans@cumin1003> START - Cookbook sre.hosts.reboot-single for host cirrussearch2111.codfw.wmnet [production]
15:28 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
15:28 <volans@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cirrussearch2111.codfw.wmnet [production]
15:15 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
15:15 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
15:12 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [production]
15:12 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/changeprop: apply [production]
15:10 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
15:10 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
15:10 <fabfur@cumin1002> conftool action : set/pooled=yes; selector: name=cp7001.magru.wmnet [production]
15:10 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
15:10 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/changeprop: apply [production]
15:10 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
15:09 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/changeprop: apply [production]
15:08 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
15:08 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
15:08 <elukey@deploy1003> helmfile [staging] DONE helmfile.d/services/kartotherian: sync [production]
15:08 <elukey@deploy1003> helmfile [staging] START helmfile.d/services/kartotherian: sync [production]
15:08 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/changeprop: apply [production]
15:08 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/changeprop: apply [production]
15:04 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp7001.magru.wmnet [production]
15:03 <fabfur> temporary depooling cp7001 to restart haproxy (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1150690) (T392219) [production]
14:56 <cgoubert@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [production]
14:56 <cgoubert@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-cron: apply [production]
14:48 <cgoubert@deploy1003> Finished scap sync-world: mediawiki-cli image update - T395245 (duration: 10m 41s) [production]
14:41 <elukey@deploy1003> helmfile [staging] DONE helmfile.d/services/kartotherian: sync [production]