|
2020-11-04
§
|
| 06:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es1027 with minimum weight after recloning T261717', diff saved to https://phabricator.wikimedia.org/P13156 and previous config saved to /var/cache/conftool/dbconfig/20201104-061829-marostegui.json |
[production] |
| 06:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es1026 with minimum weight after recloning T261717', diff saved to https://phabricator.wikimedia.org/P13155 and previous config saved to /var/cache/conftool/dbconfig/20201104-061549-marostegui.json |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1014 (re)pooling @ 25%: After cloning es1028 T261717', diff saved to https://phabricator.wikimedia.org/P13154 and previous config saved to /var/cache/conftool/dbconfig/20201104-061416-root.json |
[production] |
| 06:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1012 (re)pooling @ 25%: After cloning es1027 T261717', diff saved to https://phabricator.wikimedia.org/P13153 and previous config saved to /var/cache/conftool/dbconfig/20201104-061355-root.json |
[production] |
| 06:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1011 (re)pooling @ 25%: After cloning es1026 T261717', diff saved to https://phabricator.wikimedia.org/P13152 and previous config saved to /var/cache/conftool/dbconfig/20201104-061339-root.json |
[production] |
|
2020-11-03
§
|
| 22:56 |
<_joe_> |
repooling mw1346 |
[production] |
| 22:55 |
<_joe_> |
depooling mw1346 |
[production] |
| 22:49 |
<cdanis> |
mw1342 restart-php7.2-fpm |
[production] |
| 22:37 |
<cdanis> |
repool mw1278 and mw1279 |
[production] |
| 22:35 |
<cdanis> |
✔️ cdanis@mw1290.eqiad.wmnet ~ 🕠🍺 sudo restart-php7.2-fpm |
[production] |
| 22:34 |
<cdanis> |
restart-php7.2-fpm and pool on mw1276 |
[production] |
| 22:31 |
<cdanis> |
depool mw1276 and mw1279 also |
[production] |
| 22:25 |
<cdanis> |
✔️ cdanis@mw1278.eqiad.wmnet ~ 🕠🍺 sudo depool |
[production] |
| 21:16 |
<hashar> |
Gerrit: triggering java garbage collection # T263008 |
[production] |
| 19:32 |
<gehel> |
restarting blazegraph on wdqs1007 to reset ban list |
[production] |
| 18:21 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:15 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:45 |
<cmjohnson1> |
shutting elastic1063 down to reseat DIMM T265113 |
[production] |
| 17:32 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:31 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:31 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 17:31 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:30 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:30 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:30 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:29 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:57 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:51 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:39 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:39 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:36 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:35 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:35 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:35 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:22 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:22 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:21 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:21 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:13 |
<cdanis@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 16:13 |
<cdanis@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 16:04 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 16:03 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:01 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 16:01 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 15:59 |
<elukey> |
shutdown kafka-jumbo1006 to replace 1G with 10G nic |
[production] |
| 15:52 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:51 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:51 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:51 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:08 |
<moritzm> |
imported php-redis/xdebug to component/php72 for buster-wikimedia |
[production] |