|
2019-02-21
§
|
| 18:59 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@2d84709]: Change default task serializer of celery from pickle to json (T206333) (duration: 16m 54s) |
[production] |
| 18:46 |
<jynus> |
shutting down db1114 T214720 |
[production] |
| 18:42 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@2d84709]: Change default task serializer of celery from pickle to json (T206333) |
[production] |
| 18:33 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 18:30 |
<robh> |
ignore icinga1001 alerts, rebooting it into hardware tests via T214760 |
[production] |
| 18:29 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 18:28 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 18:28 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@5d50713]: (no justification provided) (duration: 14m 37s) |
[production] |
| 18:13 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@5d50713]: (no justification provided) |
[production] |
| 17:54 |
<robh> |
cp5007 rebooting into bios update and hardware testing via T216716 |
[production] |
| 17:47 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 17:11 |
<bblack> |
eqsin: restarting all varnish frontends to wipe cache after purge loss (site currently depooled) (skipping 5006/7 since they're being rebooted for bios flashing anyways) |
[production] |
| 17:10 |
<robh> |
rebooting cp5006 to flash bios in memory troubleshooting steps via T216717 |
[production] |
| 16:50 |
<bblack> |
eqsin: restarting all varnish backends to wipe cache after purge loss (site currently depooled) |
[production] |
| 16:41 |
<volans> |
applied hot band-aid patch to spicerack/remote.py on cumin2001 ( https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/481858 ) |
[production] |
| 16:38 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 16:23 |
<herron> |
updated phabricator.wikimedia.org spf record T216714 |
[production] |
| 16:21 |
<fsero> |
uploading scap3 3.9.0.1 package to trusty, jessie and stretch T216666 |
[production] |
| 16:20 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 16:18 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 16:17 |
<fsero> |
uploading scap3 3.9.0.1 package to trusty, jessie and stretch |
[production] |
| 16:17 |
<fsero> |
updating scap3 to 3.9.0-1 |
[production] |
| 15:57 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 15:52 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 15:23 |
<moritzm> |
installing krb5 updates for jessie |
[production] |
| 15:07 |
<herron> |
migrating ES shards away from logstash100[456] with "cluster.routing.allocation.exclude._name" : "logstash1004-production-logstash-eqiad,logstash1005-production-logstash-eqiad,logstash1006-production-logstash-eqiad” T214608 |
[production] |
| 14:50 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 14:50 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 14:41 |
<bmansurov@deploy1001> |
Finished deploy [recommendation-api/deploy@600e689]: Update to 0bb0a07626a74e0aea6dfbad669c31f76fc73365 (duration: 04m 59s) |
[production] |
| 14:37 |
<bblack> |
restart vhtcpd on cp5002 to debug multicast loss |
[production] |
| 14:36 |
<bmansurov@deploy1001> |
Started deploy [recommendation-api/deploy@600e689]: Update to 0bb0a07626a74e0aea6dfbad669c31f76fc73365 |
[production] |
| 13:57 |
<godog> |
depool and reimage logstash1007 - T213898 |
[production] |
| 13:25 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 13:20 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 16s) |
[production] |
| 13:19 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 13:19 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) |
[production] |
| 13:19 |
<jbond42> |
restarting hhvm and updateing apache on deploy1001.eqiad.wmnet |
[production] |
| 13:18 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 13:18 |
<gehel> |
restarting rolling upgrade on elasticsearch / cirrus / codfw - T215931 |
[production] |
| 12:50 |
<jbond42> |
restarting hhvm and updateing apache on mwmaint1002.eqiad.wmnet |
[production] |
| 12:44 |
<zeljkof> |
EU SWAT finished |
[production] |
| 12:42 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:491823|Add img.raremaps.com at wgCopyUploadsDomains (T216638)]] (duration: 00m 52s) |
[production] |
| 12:40 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 20s) |
[production] |
| 12:39 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 12:38 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:491826|Throttle rule for National Gallery of Canada Library and Archives edit-a-thon (T216642)]] (duration: 00m 53s) |
[production] |
| 12:33 |
<arturo> |
disable puppet in cloudnet2001-dev to test T216497 |
[production] |
| 12:31 |
<akosiaris@deploy1001> |
scap-helm mathoid finished |
[production] |
| 12:31 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster codfw completed |
[production] |
| 12:30 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster eqiad completed |
[production] |
| 12:30 |
<akosiaris@deploy1001> |
scap-helm mathoid upgrade --recreate-pods -f mathoid-values.yaml production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] |
[production] |