|
2020-11-25
§
|
| 23:28 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 22:55 |
<mutante> |
mwdebug1003 - scap pull - which rsyncs from deploy1001 and runs php-fpm restart check script (T245757) |
[production] |
| 22:47 |
<ejegg> |
increased Ingenico API call timeout |
[production] |
| 22:34 |
<shdubsh> |
beginning rolling restart of logstash cluster - eqiad |
[production] |
| 22:23 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 22:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:14 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:12 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 21:19 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 20:49 |
<krinkle@deploy1001> |
Synchronized php-1.36.0-wmf.18/includes/libs/CSSMin.php: I26ed3e5e9a - fix T268308 (duration: 00m 59s) |
[production] |
| 20:43 |
<mutante> |
LDAP added user duminasi to group wmf (T266791) |
[production] |
| 20:06 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-restart (exit_code=0) |
[production] |
| 18:44 |
<elukey> |
upload new hive* packages 2.2.3-2 to stretch-wikimedia - thirdparty/bigtop14 component |
[production] |
| 18:42 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-restart |
[production] |
| 18:38 |
<mutante> |
LDAP adding swagoel to NDA T267314#6625628 |
[production] |
| 18:31 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) |
[production] |
| 18:05 |
<ryankemper> |
T268770 [cloudelastic] Thawed writes to cloudelastic cluster following restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic --thaw` on `mwmaint1002` |
[production] |
| 18:01 |
<ryankemper> |
[cloudelastic] (forgot to mention this) Thawed writes to cloudelastic cluster following restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic --thaw` on `mwmaint1002` |
[production] |
| 17:58 |
<ryankemper> |
T268770 [cloudelastic] restarts complete, service is healthy. This is done. |
[production] |
| 17:55 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1006` complete and all 3 elasticsearch clusters are green, all cloudelastic instances are now complete |
[production] |
| 17:49 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1005` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:44 |
<shdubsh> |
beginning rolling restart of logstash cluster - codfw |
[production] |
| 17:44 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1004` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:39 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1003` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:39 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1002` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:28 |
<ryankemper> |
T268770 [cloudelastic] restarts on `cloudelastic1001` complete and all 3 elasticsearch clusters are green, proceeding to next instance |
[production] |
| 17:22 |
<ryankemper> |
T268770 Freezing writes to cloudelastic in preparation for restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic` on `mwmaint1002` |
[production] |
| 17:09 |
<ryankemper> |
T268770 [cloudelastic] Downtimed `cloudelastic100[1-6]` in icinga in preparation for cloudelastic search elasticsearch cluster restart |
[production] |
| 17:05 |
<ryankemper> |
T268770 Begin rolling restart of eqiad cirrus elasticsearch, 3 nodes at a time |
[production] |
| 17:04 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-restart |
[production] |
| 17:00 |
<godog> |
fail sdk on ms-be2031 |
[production] |
| 16:49 |
<godog> |
clean up sdk1 on / on ms-be2031 |
[production] |
| 16:46 |
<elukey> |
move analytics1066 to C3 - T267065 |
[production] |
| 16:44 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 16:21 |
<mutante> |
puppetmaster - revoking old and signing new cert for mwdebug1003 |
[production] |
| 16:11 |
<elukey> |
move analytics1065 to C3 - T267065 |
[production] |
| 16:10 |
<mutante> |
shutting down mwdebug1003 - reimaging for T245757 |
[production] |
| 16:08 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:08 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:02 |
<moritzm> |
installing golang-1.7 updates for stretch |
[production] |
| 15:57 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:57 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 15:57 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:38 |
<elukey> |
move stat1004 to A5 - T267065 |
[production] |
| 15:37 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:34 |
<moritzm> |
removing maps2002 from debmonitor |
[production] |
| 15:10 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:04 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |