951-1000 of 10000 results (99ms)
2024-07-10 ยง
14:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P66132 and previous config saved to /var/cache/conftool/dbconfig/20240710-142144-marostegui.json [production]
14:21 <kamila@deploy1002> helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [production]
14:20 <kamila@deploy1002> helmfile [codfw] START helmfile.d/services/shellbox-video: apply [production]
14:20 <kamila@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [production]
14:19 <kamila@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
14:19 <kamila@deploy1002> helmfile [staging] DONE helmfile.d/services/shellbox-video: apply [production]
14:19 <kamila@deploy1002> helmfile [staging] START helmfile.d/services/shellbox-video: apply [production]
14:16 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
14:15 <effie> disable puppet on mw memcached hosts - T352885 [production]
14:13 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
14:13 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
14:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2194 (T367856)', diff saved to https://phabricator.wikimedia.org/P66131 and previous config saved to /var/cache/conftool/dbconfig/20240710-141222-marostegui.json [production]
14:11 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
14:11 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
14:10 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
14:08 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on lsw1-e1-eqiad.mgmt with reason: prep JunOS upgrade lsw1-e1-eqiad [production]
14:08 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 1:30:00 on lsw1-e1-eqiad.mgmt with reason: prep JunOS upgrade lsw1-e1-eqiad [production]
14:07 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
14:07 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P66130 and previous config saved to /var/cache/conftool/dbconfig/20240710-140637-marostegui.json [production]
14:06 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
14:06 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
14:05 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
14:05 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
14:04 <XioNoX> add ipxe_1.21.1+git-20240627.b66e27d to bookworm-wikimedia reprepro [production]
14:04 <cgoubert@cumin1002> START - Cookbook sre.dns.netbox [production]
14:04 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
14:03 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1010.eqiad.wmnet,db1190.eqiad.wmnet,dbproxy1026.eqiad.wmnet with reason: T365993 [production]
14:02 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on backup1010.eqiad.wmnet,db1190.eqiad.wmnet,dbproxy1026.eqiad.wmnet with reason: T365993 [production]
14:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'T365993 - depool db1190 - s4', diff saved to https://phabricator.wikimedia.org/P66129 and previous config saved to /var/cache/conftool/dbconfig/20240710-140224-arnaudb.json [production]
13:57 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1172 (T367781)', diff saved to https://phabricator.wikimedia.org/P66128 and previous config saved to /var/cache/conftool/dbconfig/20240710-135656-arnaudb.json [production]
13:57 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
13:56 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on kubernetes1051.eqiad.wmnet with reason: Hardware issue [production]
13:56 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
13:56 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:56 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on kubernetes1051.eqiad.wmnet with reason: Hardware issue [production]
13:56 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T367781)', diff saved to https://phabricator.wikimedia.org/P66127 and previous config saved to /var/cache/conftool/dbconfig/20240710-135619-arnaudb.json [production]
13:54 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
13:53 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
13:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2159 (T367856)', diff saved to https://phabricator.wikimedia.org/P66126 and previous config saved to /var/cache/conftool/dbconfig/20240710-135130-marostegui.json [production]
13:49 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
13:48 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [production]
13:46 <akosiaris@cumin1002> conftool action : set/pooled=inactive; selector: name=kubernetes1059.* [production]
13:44 <btullis> re-enabling the misc dumps jobs on snapshot1017 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1053315 [production]
13:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P66125 and previous config saved to /var/cache/conftool/dbconfig/20240710-134112-arnaudb.json [production]
13:34 <klausman@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
13:34 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-mariadb1001.eqiad.wmnet with OS bookworm [production]
13:33 <klausman@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
13:26 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P66124 and previous config saved to /var/cache/conftool/dbconfig/20240710-132604-arnaudb.json [production]