|
2019-02-22
ยง
|
| 13:00 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 12:56 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 12:48 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 12:43 |
<moritzm> |
rebooting auth1002 for kernel update |
[production] |
| 12:17 |
<moritzm> |
rebooting tungsten to pick up updated microcode to address SSBD/L1TF |
[production] |
| 12:13 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 12:12 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 12:12 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 11:54 |
<moritzm> |
various reboots of servers with Westmere-EP CPUs to pick up updated microcode to address SSBD/L1TF |
[production] |
| 11:41 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 11:41 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 11:34 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 11:34 |
<moritzm> |
rebooting cp1008 for some microcode test |
[production] |
| 11:33 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 11:32 |
<jijiki> |
Pooling thumbor2002 after upgrade - T214597 |
[production] |
| 11:20 |
<moritzm> |
imported intel-microcode 3.20180807a.2 for jessie-wikimedia (T216802) |
[production] |
| 11:01 |
<godog> |
swift eqiad set thumbor write ACLs for wikipedia-meta-local-thumb |
[production] |
| 10:37 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 10:36 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 10:35 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 10:15 |
<jijiki> |
Pooling thumbor1004 after upgrade - T214597 |
[production] |
| 09:55 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 09:51 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 09:51 |
<moritzm> |
fixed package state on mw2167 |
[production] |
| 09:38 |
<akosiaris@deploy1001> |
scap-helm citoid install -n staging -f citoid-staging-values.yaml stable/citoid [namespace: citoid, clusters: staging] |
[production] |
| 09:33 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 09:33 |
<moritzm> |
installing tor security update on torrelay1001 |
[production] |
| 09:33 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 09:32 |
<_joe_> |
set pooled=inactive on mw1272, T211668 |
[production] |
| 09:26 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 09:22 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 16s) |
[production] |
| 09:22 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 09:22 |
<moritzm> |
updated tor packages to 0.3.5.8-1~d90.stretch+1 |
[production] |
| 09:18 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) |
[production] |
| 09:16 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 14s) |
[production] |
| 09:16 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 09:16 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 09:16 |
<gehel> |
starting rolling upgrade on elasticsearch / cirrus / eqiad - T215931 |
[production] |
| 08:52 |
<godog> |
force ftpsync run on sodium after debian mirror update |
[production] |
| 08:19 |
<moritzm> |
installing uriparser security updates |
[production] |
| 08:18 |
<godog> |
temporarily stop prometheus global on prometheus2004 to take a snapshot |
[production] |
| 07:47 |
<moritzm> |
installing krb5 updates for jessie |
[production] |
| 07:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool es1013 after MySQL upgrade (duration: 00m 46s) |
[production] |
| 07:28 |
<elukey> |
manually delete WANCache:v:metawiki:translate-groups from memcache on mc1022 to test fix for T203786 |
[production] |
| 07:21 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Give more traffic to es1013 after MySQL upgrade (duration: 00m 45s) |
[production] |
| 07:15 |
<_joe_> |
deactivating mw1272, memory problems |
[production] |
| 07:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool es1013 after MySQL upgrade (duration: 00m 45s) |
[production] |
| 06:51 |
<marostegui> |
Power cycle mw1272 as it crashed - T211668 |
[production] |
| 06:49 |
<marostegui> |
Stop MySQL on es1013 to upgrade MySQL |
[production] |
| 06:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool es1013 for MySQL upgrade (duration: 02m 50s) |
[production] |