6001-6050 of 10000 results (33ms)
2020-11-04 §
06:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es1028 with minimum weight after recloning T261717', diff saved to https://phabricator.wikimedia.org/P13160 and previous config saved to /var/cache/conftool/dbconfig/20201104-063028-marostegui.json [production]
06:29 <marostegui@cumin1001> dbctl commit (dc=all): 'es1014 (re)pooling @ 50%: After cloning es1028 T261717', diff saved to https://phabricator.wikimedia.org/P13159 and previous config saved to /var/cache/conftool/dbconfig/20201104-062919-root.json [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'es1012 (re)pooling @ 50%: After cloning es1027 T261717', diff saved to https://phabricator.wikimedia.org/P13158 and previous config saved to /var/cache/conftool/dbconfig/20201104-062858-root.json [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'es1011 (re)pooling @ 50%: After cloning es1026 T261717', diff saved to https://phabricator.wikimedia.org/P13157 and previous config saved to /var/cache/conftool/dbconfig/20201104-062842-root.json [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es1027 with minimum weight after recloning T261717', diff saved to https://phabricator.wikimedia.org/P13156 and previous config saved to /var/cache/conftool/dbconfig/20201104-061829-marostegui.json [production]
06:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool es1026 with minimum weight after recloning T261717', diff saved to https://phabricator.wikimedia.org/P13155 and previous config saved to /var/cache/conftool/dbconfig/20201104-061549-marostegui.json [production]
06:14 <marostegui@cumin1001> dbctl commit (dc=all): 'es1014 (re)pooling @ 25%: After cloning es1028 T261717', diff saved to https://phabricator.wikimedia.org/P13154 and previous config saved to /var/cache/conftool/dbconfig/20201104-061416-root.json [production]
06:13 <marostegui@cumin1001> dbctl commit (dc=all): 'es1012 (re)pooling @ 25%: After cloning es1027 T261717', diff saved to https://phabricator.wikimedia.org/P13153 and previous config saved to /var/cache/conftool/dbconfig/20201104-061355-root.json [production]
06:13 <marostegui@cumin1001> dbctl commit (dc=all): 'es1011 (re)pooling @ 25%: After cloning es1026 T261717', diff saved to https://phabricator.wikimedia.org/P13152 and previous config saved to /var/cache/conftool/dbconfig/20201104-061339-root.json [production]
2020-11-03 §
22:56 <_joe_> repooling mw1346 [production]
22:55 <_joe_> depooling mw1346 [production]
22:49 <cdanis> mw1342 restart-php7.2-fpm [production]
22:37 <cdanis> repool mw1278 and mw1279 [production]
22:35 <cdanis> ✔️ cdanis@mw1290.eqiad.wmnet ~ 🕠🍺 sudo restart-php7.2-fpm [production]
22:34 <cdanis> restart-php7.2-fpm and pool on mw1276 [production]
22:31 <cdanis> depool mw1276 and mw1279 also [production]
22:25 <cdanis> ✔️ cdanis@mw1278.eqiad.wmnet ~ 🕠🍺 sudo depool [production]
21:16 <hashar> Gerrit: triggering java garbage collection # T263008 [production]
19:32 <gehel> restarting blazegraph on wdqs1007 to reset ban list [production]
18:21 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:15 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
17:45 <cmjohnson1> shutting elastic1063 down to reseat DIMM T265113 [production]
17:32 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:31 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:31 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:31 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:30 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:30 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:30 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:29 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:57 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:51 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:39 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:39 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:36 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:35 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:35 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:35 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:22 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:22 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:21 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:21 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:13 <cdanis@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
16:13 <cdanis@cumin1001> START - Cookbook sre.network.cf [production]
16:04 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:03 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
16:01 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
16:01 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:59 <elukey> shutdown kafka-jumbo1006 to replace 1G with 10G nic [production]
15:52 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]