5701-5750 of 10000 results (51ms)
2022-01-25 ยง
14:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T299827)', diff saved to https://phabricator.wikimedia.org/P19168 and previous config saved to /var/cache/conftool/dbconfig/20220125-143218-marostegui.json [production]
14:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031', diff saved to https://phabricator.wikimedia.org/P19167 and previous config saved to /var/cache/conftool/dbconfig/20220125-143043-ladsgroup.json [production]
14:30 <marostegui@cumin1001> dbctl commit (dc=all): 'es1026 (re)pooling @ 5%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19166 and previous config saved to /var/cache/conftool/dbconfig/20220125-143024-root.json [production]
14:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove logpager from s8 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P19165 and previous config saved to /var/cache/conftool/dbconfig/20220125-142614-marostegui.json [production]
14:23 <jelto@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gitlab-runner1001.eqiad.wmnet on all recursors [production]
14:23 <jelto@cumin1001> START - Cookbook sre.dns.wipe-cache gitlab-runner1001.eqiad.wmnet on all recursors [production]
14:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P19164 and previous config saved to /var/cache/conftool/dbconfig/20220125-141714-marostegui.json [production]
14:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031 (T299911)', diff saved to https://phabricator.wikimedia.org/P19163 and previous config saved to /var/cache/conftool/dbconfig/20220125-141538-ladsgroup.json [production]
14:15 <marostegui@cumin1001> dbctl commit (dc=all): 'es1026 (re)pooling @ 1%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19162 and previous config saved to /var/cache/conftool/dbconfig/20220125-141520-root.json [production]
14:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1149 (T285149)', diff saved to https://phabricator.wikimedia.org/P19161 and previous config saved to /var/cache/conftool/dbconfig/20220125-141520-marostegui.json [production]
14:15 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
14:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
14:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T285149)', diff saved to https://phabricator.wikimedia.org/P19160 and previous config saved to /var/cache/conftool/dbconfig/20220125-141513-marostegui.json [production]
14:13 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1026.eqiad.wmnet with OS bullseye [production]
14:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P19159 and previous config saved to /var/cache/conftool/dbconfig/20220125-140209-marostegui.json [production]
14:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P19158 and previous config saved to /var/cache/conftool/dbconfig/20220125-140008-marostegui.json [production]
13:56 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1031.eqiad.wmnet with OS bullseye [production]
13:55 <volans@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es1022.eqiad.wmnet with OS bullseye [production]
13:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2086 (s7,s8) T299882', diff saved to https://phabricator.wikimedia.org/P19157 and previous config saved to /var/cache/conftool/dbconfig/20220125-135212-marostegui.json [production]
13:50 <volans@cumin1001> START - Cookbook sre.hosts.reimage for host es1022.eqiad.wmnet with OS bullseye [production]
13:48 <volans@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host es1022.eqiad.wmnet with OS bullseye [production]
13:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T299827)', diff saved to https://phabricator.wikimedia.org/P19156 and previous config saved to /var/cache/conftool/dbconfig/20220125-134704-marostegui.json [production]
13:46 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts gitlab-runner1001.eqiad.wmnet [production]
13:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1174 (T299827)', diff saved to https://phabricator.wikimedia.org/P19155 and previous config saved to /var/cache/conftool/dbconfig/20220125-134557-marostegui.json [production]
13:45 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
13:45 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
13:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T299827)', diff saved to https://phabricator.wikimedia.org/P19154 and previous config saved to /var/cache/conftool/dbconfig/20220125-134547-marostegui.json [production]
13:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P19153 and previous config saved to /var/cache/conftool/dbconfig/20220125-134503-marostegui.json [production]
13:43 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host es1026.eqiad.wmnet with OS bullseye [production]
13:38 <jelto@cumin1001> START - Cookbook sre.hosts.decommission for hosts gitlab-runner1001.eqiad.wmnet [production]
13:33 <_joe_> restarted pybal on lvs6003 [production]
13:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti1005.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
13:33 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti1005.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
13:31 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=drmrs,cluster=ncredir,name=ncredir6001.drmrs.wmnet [production]
13:30 <oblivian@puppetmaster1001> conftool action : set/weight=1; selector: dc=drmrs,cluster=ncredir [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P19151 and previous config saved to /var/cache/conftool/dbconfig/20220125-133042-marostegui.json [production]
13:30 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on gitlab-runner1001.eqiad.wmnet with reason: move gitlab-runner1001 to new ganeti row [production]
13:30 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on gitlab-runner1001.eqiad.wmnet with reason: move gitlab-runner1001 to new ganeti row [production]
13:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T285149)', diff saved to https://phabricator.wikimedia.org/P19150 and previous config saved to /var/cache/conftool/dbconfig/20220125-132958-marostegui.json [production]
13:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1160 (T285149)', diff saved to https://phabricator.wikimedia.org/P19149 and previous config saved to /var/cache/conftool/dbconfig/20220125-132852-marostegui.json [production]
13:28 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
13:28 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
13:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T285149)', diff saved to https://phabricator.wikimedia.org/P19148 and previous config saved to /var/cache/conftool/dbconfig/20220125-132844-marostegui.json [production]
13:27 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
13:26 <volans@cumin1001> START - Cookbook sre.hosts.reimage for host es1022.eqiad.wmnet with OS bullseye [production]
13:25 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
13:25 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
13:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.reimage for host es1031.eqiad.wmnet with OS bullseye [production]
13:22 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: sync on staging [production]
13:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]