|
2022-02-01
ยง
|
| 15:11 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti1010.eqiad.wmnet |
[production] |
| 15:10 |
<jelto> |
update scap to 4.2.2 on all hosts - T300392 |
[production] |
| 15:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P19837 and previous config saved to /var/cache/conftool/dbconfig/20220201-150818-marostegui.json |
[production] |
| 15:07 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti1016.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage |
[production] |
| 15:07 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti1016.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage |
[production] |
| 15:05 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum6002.drmrs.wmnet |
[production] |
| 15:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298558)', diff saved to https://phabricator.wikimedia.org/P19836 and previous config saved to /var/cache/conftool/dbconfig/20220201-150155-marostegui.json |
[production] |
| 15:01 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: sync on production |
[production] |
| 15:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3314 (T298558)', diff saved to https://phabricator.wikimedia.org/P19835 and previous config saved to /var/cache/conftool/dbconfig/20220201-150049-marostegui.json |
[production] |
| 15:00 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298558)', diff saved to https://phabricator.wikimedia.org/P19834 and previous config saved to /var/cache/conftool/dbconfig/20220201-150041-marostegui.json |
[production] |
| 14:59 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply on staging |
[production] |
| 14:59 |
<kartik@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply on production |
[production] |
| 14:58 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: sync on production |
[production] |
| 14:56 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply on staging |
[production] |
| 14:56 |
<kartik@deploy1002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply on production |
[production] |
| 14:53 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: sync on staging |
[production] |
| 14:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P19833 and previous config saved to /var/cache/conftool/dbconfig/20220201-145314-marostegui.json |
[production] |
| 14:52 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply on production |
[production] |
| 14:52 |
<kartik@deploy1002> |
helmfile [staging] START helmfile.d/services/cxserver: apply on staging |
[production] |
| 14:52 |
<mmandere@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host durum6002.drmrs.wmnet |
[production] |
| 14:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P19832 and previous config saved to /var/cache/conftool/dbconfig/20220201-144536-marostegui.json |
[production] |
| 14:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T300402)', diff saved to https://phabricator.wikimedia.org/P19831 and previous config saved to /var/cache/conftool/dbconfig/20220201-143809-marostegui.json |
[production] |
| 14:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3315 (T300402)', diff saved to https://phabricator.wikimedia.org/P19830 and previous config saved to /var/cache/conftool/dbconfig/20220201-143504-marostegui.json |
[production] |
| 14:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:35 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T300402)', diff saved to https://phabricator.wikimedia.org/P19829 and previous config saved to /var/cache/conftool/dbconfig/20220201-143456-marostegui.json |
[production] |
| 14:30 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve2005.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 14:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P19828 and previous config saved to /var/cache/conftool/dbconfig/20220201-143031-marostegui.json |
[production] |
| 14:21 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host ml-serve2005.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 14:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P19827 and previous config saved to /var/cache/conftool/dbconfig/20220201-141952-marostegui.json |
[production] |
| 14:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298558)', diff saved to https://phabricator.wikimedia.org/P19826 and previous config saved to /var/cache/conftool/dbconfig/20220201-141527-marostegui.json |
[production] |
| 14:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1141 (T298558)', diff saved to https://phabricator.wikimedia.org/P19825 and previous config saved to /var/cache/conftool/dbconfig/20220201-141420-marostegui.json |
[production] |
| 14:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298558)', diff saved to https://phabricator.wikimedia.org/P19824 and previous config saved to /var/cache/conftool/dbconfig/20220201-141413-marostegui.json |
[production] |
| 14:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P19823 and previous config saved to /var/cache/conftool/dbconfig/20220201-140447-marostegui.json |
[production] |
| 13:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P19822 and previous config saved to /var/cache/conftool/dbconfig/20220201-135908-marostegui.json |
[production] |
| 13:54 |
<kharlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/linkrecommendation: sync on internal |
[production] |
| 13:54 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. |
[production] |
| 13:52 |
<kharlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/linkrecommendation: sync on external |
[production] |
| 13:50 |
<kharlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply on staging |
[production] |
| 13:50 |
<kharlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/linkrecommendation: apply on internal |
[production] |
| 13:50 |
<kharlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/linkrecommendation: apply on external |
[production] |
| 13:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T300402)', diff saved to https://phabricator.wikimedia.org/P19821 and previous config saved to /var/cache/conftool/dbconfig/20220201-134942-marostegui.json |
[production] |
| 13:49 |
<kharlan@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: sync on internal |
[production] |
| 13:48 |
<btullis@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. |
[production] |
| 13:48 |
<kharlan@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: sync on external |
[production] |
| 13:47 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-druid-analytics cluster: Roll restart of jvm daemons. |
[production] |