|
2022-06-24
ยง
|
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30108 and previous config saved to /var/cache/conftool/dbconfig/20220624-075154-root.json |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30107 and previous config saved to /var/cache/conftool/dbconfig/20220624-075114-root.json |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1025 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30106 and previous config saved to /var/cache/conftool/dbconfig/20220624-075102-root.json |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30105 and previous config saved to /var/cache/conftool/dbconfig/20220624-074947-root.json |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30104 and previous config saved to /var/cache/conftool/dbconfig/20220624-074345-root.json |
[production] |
| 07:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1141 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30103 and previous config saved to /var/cache/conftool/dbconfig/20220624-074204-root.json |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30102 and previous config saved to /var/cache/conftool/dbconfig/20220624-073657-root.json |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30101 and previous config saved to /var/cache/conftool/dbconfig/20220624-073651-root.json |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30100 and previous config saved to /var/cache/conftool/dbconfig/20220624-073610-root.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1025 (re)pooling @ 75%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30099 and previous config saved to /var/cache/conftool/dbconfig/20220624-073558-root.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1141 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P30098 and previous config saved to /var/cache/conftool/dbconfig/20220624-073543-root.json |
[production] |
| 07:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 5%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30097 and previous config saved to /var/cache/conftool/dbconfig/20220624-073444-root.json |
[production] |
| 07:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ml-cache[2001-2003].codfw.wmnet with reason: reboots |
[production] |
| 07:32 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on ml-cache[2001-2003].codfw.wmnet with reason: reboots |
[production] |
| 07:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1118 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30096 and previous config saved to /var/cache/conftool/dbconfig/20220624-072841-root.json |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1118 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P30095 and previous config saved to /var/cache/conftool/dbconfig/20220624-072240-root.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30094 and previous config saved to /var/cache/conftool/dbconfig/20220624-072153-root.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30093 and previous config saved to /var/cache/conftool/dbconfig/20220624-072147-root.json |
[production] |
| 07:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1011.eqiad.wmnet |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1022 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30092 and previous config saved to /var/cache/conftool/dbconfig/20220624-072106-root.json |
[production] |
| 07:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1025 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30091 and previous config saved to /var/cache/conftool/dbconfig/20220624-072054-root.json |
[production] |
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 2%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30090 and previous config saved to /var/cache/conftool/dbconfig/20220624-071940-root.json |
[production] |
| 07:19 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host an-tool1011.eqiad.wmnet |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1096 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P30089 and previous config saved to /var/cache/conftool/dbconfig/20220624-071551-root.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P30088 and previous config saved to /var/cache/conftool/dbconfig/20220624-071439-root.json |
[production] |
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1022 es1025 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P30087 and previous config saved to /var/cache/conftool/dbconfig/20220624-070700-root.json |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30086 and previous config saved to /var/cache/conftool/dbconfig/20220624-070601-root.json |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30085 and previous config saved to /var/cache/conftool/dbconfig/20220624-070555-root.json |
[production] |
| 07:02 |
<marostegui> |
Reboot db1117 for kernel upgrade (expect haproxy irc alerts) |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30084 and previous config saved to /var/cache/conftool/dbconfig/20220624-070201-root.json |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30083 and previous config saved to /var/cache/conftool/dbconfig/20220624-070157-root.json |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1172 (re)pooling @ 100%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30082 and previous config saved to /var/cache/conftool/dbconfig/20220624-070151-root.json |
[production] |
| 06:53 |
<jynus> |
restarting bacula director @ backup1001 |
[production] |
| 06:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30081 and previous config saved to /var/cache/conftool/dbconfig/20220624-065057-root.json |
[production] |
| 06:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30080 and previous config saved to /var/cache/conftool/dbconfig/20220624-065051-root.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30079 and previous config saved to /var/cache/conftool/dbconfig/20220624-064657-root.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30078 and previous config saved to /var/cache/conftool/dbconfig/20220624-064653-root.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1172 (re)pooling @ 75%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30077 and previous config saved to /var/cache/conftool/dbconfig/20220624-064647-root.json |
[production] |
| 06:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30076 and previous config saved to /var/cache/conftool/dbconfig/20220624-063553-root.json |
[production] |
| 06:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30075 and previous config saved to /var/cache/conftool/dbconfig/20220624-063547-root.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30074 and previous config saved to /var/cache/conftool/dbconfig/20220624-063154-root.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30073 and previous config saved to /var/cache/conftool/dbconfig/20220624-063149-root.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1172 (re)pooling @ 50%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30072 and previous config saved to /var/cache/conftool/dbconfig/20220624-063143-root.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30071 and previous config saved to /var/cache/conftool/dbconfig/20220624-062049-root.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30070 and previous config saved to /var/cache/conftool/dbconfig/20220624-062043-root.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30069 and previous config saved to /var/cache/conftool/dbconfig/20220624-061650-root.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30068 and previous config saved to /var/cache/conftool/dbconfig/20220624-061645-root.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1172 (re)pooling @ 25%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30067 and previous config saved to /var/cache/conftool/dbconfig/20220624-061640-root.json |
[production] |
| 06:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30066 and previous config saved to /var/cache/conftool/dbconfig/20220624-060545-root.json |
[production] |
| 06:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 10%: After kernel upgrade', diff saved to https://phabricator.wikimedia.org/P30065 and previous config saved to /var/cache/conftool/dbconfig/20220624-060539-root.json |
[production] |