|
2019-02-21
§
|
| 16:18 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 16:17 |
<fsero> |
uploading scap3 3.9.0.1 package to trusty, jessie and stretch |
[production] |
| 16:17 |
<fsero> |
updating scap3 to 3.9.0-1 |
[production] |
| 15:57 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 15:52 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 15:23 |
<moritzm> |
installing krb5 updates for jessie |
[production] |
| 15:07 |
<herron> |
migrating ES shards away from logstash100[456] with "cluster.routing.allocation.exclude._name" : "logstash1004-production-logstash-eqiad,logstash1005-production-logstash-eqiad,logstash1006-production-logstash-eqiad” T214608 |
[production] |
| 14:50 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 14:50 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 14:41 |
<bmansurov@deploy1001> |
Finished deploy [recommendation-api/deploy@600e689]: Update to 0bb0a07626a74e0aea6dfbad669c31f76fc73365 (duration: 04m 59s) |
[production] |
| 14:37 |
<bblack> |
restart vhtcpd on cp5002 to debug multicast loss |
[production] |
| 14:36 |
<bmansurov@deploy1001> |
Started deploy [recommendation-api/deploy@600e689]: Update to 0bb0a07626a74e0aea6dfbad669c31f76fc73365 |
[production] |
| 13:57 |
<godog> |
depool and reimage logstash1007 - T213898 |
[production] |
| 13:25 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 13:20 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 16s) |
[production] |
| 13:19 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 13:19 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) |
[production] |
| 13:19 |
<jbond42> |
restarting hhvm and updateing apache on deploy1001.eqiad.wmnet |
[production] |
| 13:18 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 13:18 |
<gehel> |
restarting rolling upgrade on elasticsearch / cirrus / codfw - T215931 |
[production] |
| 12:50 |
<jbond42> |
restarting hhvm and updateing apache on mwmaint1002.eqiad.wmnet |
[production] |
| 12:44 |
<zeljkof> |
EU SWAT finished |
[production] |
| 12:42 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:491823|Add img.raremaps.com at wgCopyUploadsDomains (T216638)]] (duration: 00m 52s) |
[production] |
| 12:40 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 20s) |
[production] |
| 12:39 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 12:38 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:491826|Throttle rule for National Gallery of Canada Library and Archives edit-a-thon (T216642)]] (duration: 00m 53s) |
[production] |
| 12:33 |
<arturo> |
disable puppet in cloudnet2001-dev to test T216497 |
[production] |
| 12:31 |
<akosiaris@deploy1001> |
scap-helm mathoid finished |
[production] |
| 12:31 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster codfw completed |
[production] |
| 12:30 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster eqiad completed |
[production] |
| 12:30 |
<akosiaris@deploy1001> |
scap-helm mathoid upgrade --recreate-pods -f mathoid-values.yaml production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] |
[production] |
| 12:27 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 38s) |
[production] |
| 12:26 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 12:24 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: l thumbor2002.codfw.wmnet (duration: 00m 04s) |
[production] |
| 12:24 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: l thumbor2002.codfw.wmnet |
[production] |
| 12:24 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: l thumbor2002 (duration: 00m 08s) |
[production] |
| 12:24 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: l thumbor2002 |
[production] |
| 12:23 |
<arturo> |
importing openstack mitaka packages to reprepro @ install1002 (T216497) |
[production] |
| 12:17 |
<arturo> |
enable puppet in install1002 (done testing T216497) |
[production] |
| 12:14 |
<zfilipin@deploy1001> |
Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: [[gerrit:491699|Disable mobile main page special casing on huwiki (T216563)]] (duration: 00m 54s) |
[production] |
| 12:13 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: Updating repo (duration: 00m 29s) |
[production] |
| 12:13 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: Updating repo |
[production] |
| 12:10 |
<arturo> |
T216497 import reprepro key 7638D0442B90D010 (debian archive automatic signing key (8/jessie) |
[production] |
| 12:00 |
<arturo> |
disable puppet in install1002 to test T216497 |
[production] |
| 11:13 |
<volans> |
upgraded spicerack to 0.0.19 on cumin[12]001 |
[production] |
| 11:11 |
<volans> |
uploaded spicerack_0.0.19-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 10:55 |
<akosiaris> |
upgrade mathoid staging+production to latest helm chart |
[production] |
| 10:47 |
<akosiaris@deploy1001> |
scap-helm mathoid finished |
[production] |
| 10:47 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster codfw completed |
[production] |
| 10:47 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster eqiad completed |
[production] |