3151-3200 of 10000 results (79ms)
2022-11-10 ยง
13:36 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
13:35 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38970 and previous config saved to /var/cache/conftool/dbconfig/20221110-133040-marostegui.json [production]
13:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38969 and previous config saved to /var/cache/conftool/dbconfig/20221110-132622-ladsgroup.json [production]
13:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38968 and previous config saved to /var/cache/conftool/dbconfig/20221110-132300-marostegui.json [production]
13:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1136 (T322618)', diff saved to https://phabricator.wikimedia.org/P38967 and previous config saved to /var/cache/conftool/dbconfig/20221110-132010-ladsgroup.json [production]
13:20 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
13:19 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
13:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38966 and previous config saved to /var/cache/conftool/dbconfig/20221110-131949-ladsgroup.json [production]
13:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1107', diff saved to https://phabricator.wikimedia.org/P38965 and previous config saved to /var/cache/conftool/dbconfig/20221110-131533-marostegui.json [production]
13:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38964 and previous config saved to /var/cache/conftool/dbconfig/20221110-130753-marostegui.json [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1185 (T321130)', diff saved to https://phabricator.wikimedia.org/P38963 and previous config saved to /var/cache/conftool/dbconfig/20221110-130527-marostegui.json [production]
13:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance [production]
13:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1185.eqiad.wmnet with reason: Maintenance [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T321130)', diff saved to https://phabricator.wikimedia.org/P38962 and previous config saved to /var/cache/conftool/dbconfig/20221110-130506-marostegui.json [production]
13:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38961 and previous config saved to /var/cache/conftool/dbconfig/20221110-130443-ladsgroup.json [production]
13:04 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D [production]
13:04 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1034.eqiad.wmnet to cluster eqiad and group D [production]
13:01 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet [production]
13:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38960 and previous config saved to /var/cache/conftool/dbconfig/20221110-130027-marostegui.json [production]
12:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1107 (T321123)', diff saved to https://phabricator.wikimedia.org/P38959 and previous config saved to /var/cache/conftool/dbconfig/20221110-125919-marostegui.json [production]
12:59 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance [production]
12:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1107.eqiad.wmnet with reason: Maintenance [production]
12:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321123)', diff saved to https://phabricator.wikimedia.org/P38958 and previous config saved to /var/cache/conftool/dbconfig/20221110-125858-marostegui.json [production]
12:53 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet [production]
12:51 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
12:51 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
12:51 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
12:51 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
12:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38957 and previous config saved to /var/cache/conftool/dbconfig/20221110-124959-marostegui.json [production]
12:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P38956 and previous config saved to /var/cache/conftool/dbconfig/20221110-124936-ladsgroup.json [production]
12:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2119 (T318605)', diff saved to https://phabricator.wikimedia.org/P38955 and previous config saved to /var/cache/conftool/dbconfig/20221110-124805-ladsgroup.json [production]
12:47 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance [production]
12:47 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance [production]
12:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2110 (T318605)', diff saved to https://phabricator.wikimedia.org/P38954 and previous config saved to /var/cache/conftool/dbconfig/20221110-124743-ladsgroup.json [production]
12:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38953 and previous config saved to /var/cache/conftool/dbconfig/20221110-124352-marostegui.json [production]
12:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T322618)', diff saved to https://phabricator.wikimedia.org/P38952 and previous config saved to /var/cache/conftool/dbconfig/20221110-123527-ladsgroup.json [production]
12:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38951 and previous config saved to /var/cache/conftool/dbconfig/20221110-123453-marostegui.json [production]
12:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38950 and previous config saved to /var/cache/conftool/dbconfig/20221110-123428-ladsgroup.json [production]
12:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2110', diff saved to https://phabricator.wikimedia.org/P38949 and previous config saved to /var/cache/conftool/dbconfig/20221110-123237-ladsgroup.json [production]
12:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1127 (T322618)', diff saved to https://phabricator.wikimedia.org/P38948 and previous config saved to /var/cache/conftool/dbconfig/20221110-123215-ladsgroup.json [production]
12:32 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
12:32 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
12:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P38947 and previous config saved to /var/cache/conftool/dbconfig/20221110-123153-ladsgroup.json [production]
12:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P38946 and previous config saved to /var/cache/conftool/dbconfig/20221110-122845-marostegui.json [production]
12:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1144:3314 (T318605)', diff saved to https://phabricator.wikimedia.org/P38945 and previous config saved to /var/cache/conftool/dbconfig/20221110-122720-ladsgroup.json [production]
12:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
12:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
12:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1143 (T318605)', diff saved to https://phabricator.wikimedia.org/P38944 and previous config saved to /var/cache/conftool/dbconfig/20221110-122708-ladsgroup.json [production]
12:26 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]