3751-3800 of 10000 results (42ms)
2022-02-01 ยง
15:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P19837 and previous config saved to /var/cache/conftool/dbconfig/20220201-150818-marostegui.json [production]
15:07 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti1016.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
15:07 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti1016.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage [production]
15:05 <mmandere@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum6002.drmrs.wmnet [production]
15:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298558)', diff saved to https://phabricator.wikimedia.org/P19836 and previous config saved to /var/cache/conftool/dbconfig/20220201-150155-marostegui.json [production]
15:01 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: sync on production [production]
15:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1144:3314 (T298558)', diff saved to https://phabricator.wikimedia.org/P19835 and previous config saved to /var/cache/conftool/dbconfig/20220201-150049-marostegui.json [production]
15:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
15:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
15:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298558)', diff saved to https://phabricator.wikimedia.org/P19834 and previous config saved to /var/cache/conftool/dbconfig/20220201-150041-marostegui.json [production]
14:59 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply on staging [production]
14:59 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply on production [production]
14:58 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: sync on production [production]
14:56 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply on staging [production]
14:56 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply on production [production]
14:53 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: sync on staging [production]
14:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P19833 and previous config saved to /var/cache/conftool/dbconfig/20220201-145314-marostegui.json [production]
14:52 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply on production [production]
14:52 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply on staging [production]
14:52 <mmandere@cumin1001> START - Cookbook sre.ganeti.makevm for new host durum6002.drmrs.wmnet [production]
14:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P19832 and previous config saved to /var/cache/conftool/dbconfig/20220201-144536-marostegui.json [production]
14:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T300402)', diff saved to https://phabricator.wikimedia.org/P19831 and previous config saved to /var/cache/conftool/dbconfig/20220201-143809-marostegui.json [production]
14:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1144:3315 (T300402)', diff saved to https://phabricator.wikimedia.org/P19830 and previous config saved to /var/cache/conftool/dbconfig/20220201-143504-marostegui.json [production]
14:35 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
14:35 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
14:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T300402)', diff saved to https://phabricator.wikimedia.org/P19829 and previous config saved to /var/cache/conftool/dbconfig/20220201-143456-marostegui.json [production]
14:30 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P19828 and previous config saved to /var/cache/conftool/dbconfig/20220201-143031-marostegui.json [production]
14:21 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-serve2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P19827 and previous config saved to /var/cache/conftool/dbconfig/20220201-141952-marostegui.json [production]
14:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298558)', diff saved to https://phabricator.wikimedia.org/P19826 and previous config saved to /var/cache/conftool/dbconfig/20220201-141527-marostegui.json [production]
14:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1141 (T298558)', diff saved to https://phabricator.wikimedia.org/P19825 and previous config saved to /var/cache/conftool/dbconfig/20220201-141420-marostegui.json [production]
14:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
14:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance [production]
14:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298558)', diff saved to https://phabricator.wikimedia.org/P19824 and previous config saved to /var/cache/conftool/dbconfig/20220201-141413-marostegui.json [production]
14:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P19823 and previous config saved to /var/cache/conftool/dbconfig/20220201-140447-marostegui.json [production]
13:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P19822 and previous config saved to /var/cache/conftool/dbconfig/20220201-135908-marostegui.json [production]
13:54 <kharlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/linkrecommendation: sync on internal [production]
13:54 <btullis@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [production]
13:52 <kharlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/linkrecommendation: sync on external [production]
13:50 <kharlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply on staging [production]
13:50 <kharlan@deploy1002> helmfile [codfw] START helmfile.d/services/linkrecommendation: apply on internal [production]
13:50 <kharlan@deploy1002> helmfile [codfw] START helmfile.d/services/linkrecommendation: apply on external [production]
13:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T300402)', diff saved to https://phabricator.wikimedia.org/P19821 and previous config saved to /var/cache/conftool/dbconfig/20220201-134942-marostegui.json [production]
13:49 <kharlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: sync on internal [production]
13:48 <btullis@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [production]
13:48 <kharlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: sync on external [production]
13:47 <btullis@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-druid-analytics cluster: Roll restart of jvm daemons. [production]
13:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1161 (T300402)', diff saved to https://phabricator.wikimedia.org/P19820 and previous config saved to /var/cache/conftool/dbconfig/20220201-134740-marostegui.json [production]
13:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]