|
2022-09-10
§
|
| 02:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1141 (T314041)', diff saved to https://phabricator.wikimedia.org/P34366 and previous config saved to /var/cache/conftool/dbconfig/20220910-024401-ladsgroup.json |
[production] |
| 02:43 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:43 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121 (T314041)', diff saved to https://phabricator.wikimedia.org/P34365 and previous config saved to /var/cache/conftool/dbconfig/20220910-024339-ladsgroup.json |
[production] |
| 02:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P34364 and previous config saved to /var/cache/conftool/dbconfig/20220910-024019-ladsgroup.json |
[production] |
| 02:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P34363 and previous config saved to /var/cache/conftool/dbconfig/20220910-022833-ladsgroup.json |
[production] |
| 02:25 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P34362 and previous config saved to /var/cache/conftool/dbconfig/20220910-022513-ladsgroup.json |
[production] |
| 02:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P34361 and previous config saved to /var/cache/conftool/dbconfig/20220910-021326-ladsgroup.json |
[production] |
| 02:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T312863)', diff saved to https://phabricator.wikimedia.org/P34360 and previous config saved to /var/cache/conftool/dbconfig/20220910-021007-ladsgroup.json |
[production] |
| 01:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1121 (T314041)', diff saved to https://phabricator.wikimedia.org/P34359 and previous config saved to /var/cache/conftool/dbconfig/20220910-015820-ladsgroup.json |
[production] |
| 00:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2137:3314 (T314041)', diff saved to https://phabricator.wikimedia.org/P34358 and previous config saved to /var/cache/conftool/dbconfig/20220910-005046-ladsgroup.json |
[production] |
| 00:50 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 00:50 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance |
[production] |
| 00:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2106 (T314041)', diff saved to https://phabricator.wikimedia.org/P34357 and previous config saved to /var/cache/conftool/dbconfig/20220910-005025-ladsgroup.json |
[production] |
| 00:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P34356 and previous config saved to /var/cache/conftool/dbconfig/20220910-003518-ladsgroup.json |
[production] |
| 00:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P34355 and previous config saved to /var/cache/conftool/dbconfig/20220910-002012-ladsgroup.json |
[production] |
| 00:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2106 (T314041)', diff saved to https://phabricator.wikimedia.org/P34354 and previous config saved to /var/cache/conftool/dbconfig/20220910-000504-ladsgroup.json |
[production] |
|
2022-09-09
§
|
| 22:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1184 (T312863)', diff saved to https://phabricator.wikimedia.org/P34353 and previous config saved to /var/cache/conftool/dbconfig/20220909-224245-ladsgroup.json |
[production] |
| 22:42 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:42 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1118 (T312863)', diff saved to https://phabricator.wikimedia.org/P34352 and previous config saved to /var/cache/conftool/dbconfig/20220909-224223-ladsgroup.json |
[production] |
| 22:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P34351 and previous config saved to /var/cache/conftool/dbconfig/20220909-222717-ladsgroup.json |
[production] |
| 22:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P34350 and previous config saved to /var/cache/conftool/dbconfig/20220909-221210-ladsgroup.json |
[production] |
| 21:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1118 (T312863)', diff saved to https://phabricator.wikimedia.org/P34349 and previous config saved to /var/cache/conftool/dbconfig/20220909-215704-ladsgroup.json |
[production] |
| 20:27 |
<herron@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=93) for new host dispatch-be1001.eqiad.wmnet |
[production] |
| 20:27 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dispatch-be1001.eqiad.wmnet on all recursors |
[production] |
| 20:27 |
<herron@cumin1001> |
START - Cookbook sre.dns.wipe-cache dispatch-be1001.eqiad.wmnet on all recursors |
[production] |
| 20:27 |
<herron@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 20:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2116 (T312863)', diff saved to https://phabricator.wikimedia.org/P34348 and previous config saved to /var/cache/conftool/dbconfig/20220909-201857-ladsgroup.json |
[production] |
| 20:18 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 20:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 20:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2112 (T312863)', diff saved to https://phabricator.wikimedia.org/P34347 and previous config saved to /var/cache/conftool/dbconfig/20220909-201835-ladsgroup.json |
[production] |
| 20:03 |
<herron@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:03 |
<herron@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host dispatch-be1001.eqiad.wmnet |
[production] |
| 20:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P34345 and previous config saved to /var/cache/conftool/dbconfig/20220909-200329-ladsgroup.json |
[production] |
| 20:02 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons. |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P34344 and previous config saved to /var/cache/conftool/dbconfig/20220909-194822-ladsgroup.json |
[production] |
| 19:46 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 19:45 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 19:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2112 (T312863)', diff saved to https://phabricator.wikimedia.org/P34343 and previous config saved to /var/cache/conftool/dbconfig/20220909-193316-ladsgroup.json |
[production] |
| 18:21 |
<herron@cumin1001> |
START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons. |
[production] |
| 16:21 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons. |
[production] |
| 15:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1118 (T312863)', diff saved to https://phabricator.wikimedia.org/P34342 and previous config saved to /var/cache/conftool/dbconfig/20220909-155234-ladsgroup.json |
[production] |
| 15:52 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:52 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106 (T312863)', diff saved to https://phabricator.wikimedia.org/P34341 and previous config saved to /var/cache/conftool/dbconfig/20220909-155213-ladsgroup.json |
[production] |
| 15:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P34340 and previous config saved to /var/cache/conftool/dbconfig/20220909-153706-ladsgroup.json |
[production] |
| 15:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P34339 and previous config saved to /var/cache/conftool/dbconfig/20220909-152159-ladsgroup.json |
[production] |