|
2020-08-12
ยง
|
| 18:02 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 18:02 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 18:00 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:59 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 17:58 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 17:56 |
<otto@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 17:52 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 17:52 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 17:52 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:51 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 17:51 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 17:51 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:50 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:49 |
<effie> |
reboot mw1265 mw1282 mw1283 |
[production] |
| 17:45 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:45 |
<otto@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 17:37 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) |
[production] |
| 17:36 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) |
[production] |
| 17:30 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 17:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 17:21 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:21 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:20 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:20 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:19 |
<effie> |
reboot mw1263 mw1264 mw1279 and mw1281 |
[production] |
| 17:17 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 17:17 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 17:16 |
<cdanis> |
for posterity: mw1359 has a bunch of special packages installed (previously recorded in SAL) and also has `sudo memleak-bpfcc -o 60000 -z 31 -Z 33 30` running in a tmux in an attempt to understand what's causing the page fragmentation in the appserver fleet |
[production] |
| 17:16 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 17:16 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 17:15 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 17:15 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 17:13 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . |
[production] |
| 17:13 |
<otto@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 17:03 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) |
[production] |
| 17:00 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) |
[production] |
| 16:57 |
<sbassett@deploy1001> |
Synchronized private/PrivateSettings.php: Additional mitigations for T257687 (duration: 01m 03s) |
[production] |
| 16:53 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:52 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:52 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 16:50 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:49 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 16:48 |
<oblivian@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) |
[production] |
| 16:45 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:45 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:44 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:35 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:35 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:32 |
<oblivian@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 16:31 |
<effie> |
reboot mw1277 mw1278 && mw1261 mw1262 |
[production] |