|
2020-11-25
ยง
|
| 18:38 |
<mutante> |
LDAP adding swagoel to NDA T267314#6625628 |
[production] |
| 18:31 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) |
[production] |
| 18:05 |
<ryankemper> |
T268770 [cloudelastic] Thawed writes to cloudelastic cluster following restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic --thaw` on `mwmaint1002` |
[production] |
| 18:01 |
<ryankemper> |
[cloudelastic] (forgot to mention this) Thawed writes to cloudelastic cluster following restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic --thaw` on `mwmaint1002` |
[production] |
| 17:58 |
<ryankemper> |
T268770 [cloudelastic] restarts complete, service is healthy. This is done. |
[production] |
| 17:55 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1006` complete and all 3 elasticsearch clusters are green, all cloudelastic instances are now complete |
[production] |
| 17:49 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1005` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:44 |
<shdubsh> |
beginning rolling restart of logstash cluster - codfw |
[production] |
| 17:44 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1004` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:39 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1003` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:39 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1002` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:28 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1001` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:22 |
<ryankemper> |
T268770 Freezing writes to cloudelastic in preparation for restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic` on `mwmaint1002` |
[production] |
| 17:09 |
<ryankemper> |
T268770 [cloudelastic] Downtimed `cloudelastic100[1-6]` in icinga in preparation for cloudelastic search elasticsearch cluster restart |
[production] |
| 17:05 |
<ryankemper> |
T268770 Begin rolling restart of eqiad cirrus elasticsearch, 3 nodes at a time |
[production] |
| 17:04 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-restart |
[production] |
| 17:00 |
<godog> |
fail sdk on ms-be2031 |
[production] |
| 16:49 |
<godog> |
clean up sdk1 on / on ms-be2031 |
[production] |
| 16:46 |
<elukey> |
move analytics1066 to C3 - T267065 |
[production] |
| 16:44 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 16:21 |
<mutante> |
puppetmaster - revoking old and signing new cert for mwdebug1003 |
[production] |
| 16:11 |
<elukey> |
move analytics1065 to C3 - T267065 |
[production] |
| 16:10 |
<mutante> |
shutting down mwdebug1003 - reimaging for T245757 |
[production] |
| 16:08 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:08 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:02 |
<moritzm> |
installing golang-1.7 updates for stretch |
[production] |
| 15:57 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:57 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 15:57 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:38 |
<elukey> |
move stat1004 to A5 - T267065 |
[production] |
| 15:37 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:34 |
<moritzm> |
removing maps2002 from debmonitor |
[production] |
| 15:10 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:04 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 15:04 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:03 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 14:56 |
<moritzm> |
installing krb5 security updates for Buster |
[production] |
| 14:55 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 14:55 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 14:55 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 14:26 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 14:00 |
<akosiaris@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:56 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 13:44 |
<akosiaris@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:43 |
<akosiaris> |
assign IPs to kubestage200{1,2,3}.codfw.wmnet, kubestagemaster2001.codfw.wmnet in netbox T268747 |
[production] |
| 13:14 |
<marostegui> |
Deploy schema change on commonswiki.watchlist on s4 codfw - there will be lag on s4 codfw - T268004 |
[production] |
| 13:08 |
<akosiaris> |
assign IPs to kubestage200{1,2,3}.codfw.wmnet, kubestagemaster2001.codfw.wmnet in netbox |
[production] |
| 12:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13414 and previous config saved to /var/cache/conftool/dbconfig/20201125-124202-root.json |
[production] |
| 12:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13413 and previous config saved to /var/cache/conftool/dbconfig/20201125-122659-root.json |
[production] |
| 12:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13412 and previous config saved to /var/cache/conftool/dbconfig/20201125-121155-root.json |
[production] |