|
2022-11-28
ยง
|
| 23:58 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1197 (T323827)', diff saved to https://phabricator.wikimedia.org/P41504 and previous config saved to /var/cache/conftool/dbconfig/20221128-235817-ladsgroup.json |
[production] |
| 23:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T323907)', diff saved to https://phabricator.wikimedia.org/P41503 and previous config saved to /var/cache/conftool/dbconfig/20221128-235223-ladsgroup.json |
[production] |
| 23:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P41502 and previous config saved to /var/cache/conftool/dbconfig/20221128-234834-ladsgroup.json |
[production] |
| 23:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P41501 and previous config saved to /var/cache/conftool/dbconfig/20221128-234636-ladsgroup.json |
[production] |
| 23:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P41500 and previous config saved to /var/cache/conftool/dbconfig/20221128-234535-marostegui.json |
[production] |
| 23:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P41499 and previous config saved to /var/cache/conftool/dbconfig/20221128-234311-ladsgroup.json |
[production] |
| 23:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P41498 and previous config saved to /var/cache/conftool/dbconfig/20221128-233328-ladsgroup.json |
[production] |
| 23:33 |
<ebernhardson@deploy1002> |
Finished deploy [search/mjolnir/deploy@d361052]: msearch_daemon: Remove cluster selection/load monitor (duration: 00m 51s) |
[production] |
| 23:32 |
<ebernhardson@deploy1002> |
Started deploy [search/mjolnir/deploy@d361052]: msearch_daemon: Remove cluster selection/load monitor |
[production] |
| 23:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P41497 and previous config saved to /var/cache/conftool/dbconfig/20221128-233130-ladsgroup.json |
[production] |
| 23:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2172 (T321126)', diff saved to https://phabricator.wikimedia.org/P41496 and previous config saved to /var/cache/conftool/dbconfig/20221128-233028-marostegui.json |
[production] |
| 23:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2172 (T321126)', diff saved to https://phabricator.wikimedia.org/P41495 and previous config saved to /var/cache/conftool/dbconfig/20221128-232815-marostegui.json |
[production] |
| 23:28 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance |
[production] |
| 23:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P41494 and previous config saved to /var/cache/conftool/dbconfig/20221128-232805-ladsgroup.json |
[production] |
| 23:28 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db2172.codfw.wmnet with reason: Maintenance |
[production] |
| 23:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321126)', diff saved to https://phabricator.wikimedia.org/P41493 and previous config saved to /var/cache/conftool/dbconfig/20221128-232754-marostegui.json |
[production] |
| 23:23 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@f68dc24]: deploy config changes for mysql-port-as-string (T280597) (duration: 00m 55s) |
[production] |
| 23:22 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@f68dc24]: deploy config changes for mysql-port-as-string (T280597) |
[production] |
| 23:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P41492 and previous config saved to /var/cache/conftool/dbconfig/20221128-231821-ladsgroup.json |
[production] |
| 23:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105 (T323907)', diff saved to https://phabricator.wikimedia.org/P41491 and previous config saved to /var/cache/conftool/dbconfig/20221128-231623-ladsgroup.json |
[production] |
| 23:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T323907)', diff saved to https://phabricator.wikimedia.org/P41490 and previous config saved to /var/cache/conftool/dbconfig/20221128-231548-ladsgroup.json |
[production] |
| 23:15 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1112 (T322618)', diff saved to https://phabricator.wikimedia.org/P41489 and previous config saved to /var/cache/conftool/dbconfig/20221128-231426-ladsgroup.json |
[production] |
| 23:14 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:14 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:14 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:14 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1197 (T323827)', diff saved to https://phabricator.wikimedia.org/P41488 and previous config saved to /var/cache/conftool/dbconfig/20221128-231258-ladsgroup.json |
[production] |
| 23:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P41487 and previous config saved to /var/cache/conftool/dbconfig/20221128-231247-marostegui.json |
[production] |
| 23:12 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 23:12 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20221128-225741-marostegui.json |
[production] |
| 22:56 |
<sukhe@cumin2002> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts cp5006.eqsin.wmnet |
[production] |
| 22:56 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:56 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp5006.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" |
[production] |
| 22:54 |
<sukhe@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp5006.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" |
[production] |
| 22:54 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@f68dc24]: deploy config changes for phab1001 -> phab1004 (T280597) (duration: 00m 52s) |
[production] |
| 22:53 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@f68dc24]: deploy config changes for phab1001 -> phab1004 (T280597) |
[production] |
| 22:52 |
<sukhe@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2105 (T323907)', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20221128-225101-ladsgroup.json |
[production] |
| 22:50 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance |
[production] |
| 22:50 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance |
[production] |
| 22:47 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts cp5006.eqsin.wmnet |
[production] |
| 22:42 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5006.eqsin.wmnet with reason: downtimed, to be depooled |
[production] |
| 22:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2155 (T321126)', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20221128-224235-marostegui.json |
[production] |
| 22:42 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cp5006.eqsin.wmnet with reason: downtimed, to be depooled |
[production] |
| 22:42 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp5006.eqsin.wmnet,service=varnish-fe |
[production] |
| 22:42 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp5006.eqsin.wmnet,service=ats-be |
[production] |