|
2019-02-21
§
|
| 20:35 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0) |
[production] |
| 20:18 |
<thcipriani@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.18 |
[production] |
| 20:06 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@5d937b1]: Drop accepting pickle altogether (T206333) (duration: 13m 17s) |
[production] |
| 19:58 |
<bblack> |
eqsin: repooling user traffic |
[production] |
| 19:52 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@5d937b1]: Drop accepting pickle altogether (T206333) |
[production] |
| 19:35 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:491506|Drop obsolete Wikibase configs (T213713)]], Part II (duration: 00m 53s) |
[production] |
| 19:33 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:491506|Drop obsolete Wikibase configs (T213713)]], Part I (duration: 00m 52s) |
[production] |
| 19:32 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 19:32 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 19:25 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 19:19 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT [[gerrit:491484|Set wmgWikibaseRepoIdGeneratorSeparateDbConnection to true for wikidata (T215147)]] (duration: 00m 56s) |
[production] |
| 18:59 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@2d84709]: Change default task serializer of celery from pickle to json (T206333) (duration: 16m 54s) |
[production] |
| 18:46 |
<jynus> |
shutting down db1114 T214720 |
[production] |
| 18:42 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@2d84709]: Change default task serializer of celery from pickle to json (T206333) |
[production] |
| 18:33 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 18:30 |
<robh> |
ignore icinga1001 alerts, rebooting it into hardware tests via T214760 |
[production] |
| 18:29 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 18:28 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 18:28 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@5d50713]: (no justification provided) (duration: 14m 37s) |
[production] |
| 18:13 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@5d50713]: (no justification provided) |
[production] |
| 17:54 |
<robh> |
cp5007 rebooting into bios update and hardware testing via T216716 |
[production] |
| 17:47 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 17:11 |
<bblack> |
eqsin: restarting all varnish frontends to wipe cache after purge loss (site currently depooled) (skipping 5006/7 since they're being rebooted for bios flashing anyways) |
[production] |
| 17:10 |
<robh> |
rebooting cp5006 to flash bios in memory troubleshooting steps via T216717 |
[production] |
| 16:50 |
<bblack> |
eqsin: restarting all varnish backends to wipe cache after purge loss (site currently depooled) |
[production] |
| 16:41 |
<volans> |
applied hot band-aid patch to spicerack/remote.py on cumin2001 ( https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/481858 ) |
[production] |
| 16:38 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 16:23 |
<herron> |
updated phabricator.wikimedia.org spf record T216714 |
[production] |
| 16:21 |
<fsero> |
uploading scap3 3.9.0.1 package to trusty, jessie and stretch T216666 |
[production] |
| 16:20 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 16:18 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 16:17 |
<fsero> |
uploading scap3 3.9.0.1 package to trusty, jessie and stretch |
[production] |
| 16:17 |
<fsero> |
updating scap3 to 3.9.0-1 |
[production] |
| 15:57 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 15:52 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) |
[production] |
| 15:23 |
<moritzm> |
installing krb5 updates for jessie |
[production] |
| 15:07 |
<herron> |
migrating ES shards away from logstash100[456] with "cluster.routing.allocation.exclude._name" : "logstash1004-production-logstash-eqiad,logstash1005-production-logstash-eqiad,logstash1006-production-logstash-eqiad” T214608 |
[production] |
| 14:50 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 14:50 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 14:41 |
<bmansurov@deploy1001> |
Finished deploy [recommendation-api/deploy@600e689]: Update to 0bb0a07626a74e0aea6dfbad669c31f76fc73365 (duration: 04m 59s) |
[production] |
| 14:37 |
<bblack> |
restart vhtcpd on cp5002 to debug multicast loss |
[production] |
| 14:36 |
<bmansurov@deploy1001> |
Started deploy [recommendation-api/deploy@600e689]: Update to 0bb0a07626a74e0aea6dfbad669c31f76fc73365 |
[production] |
| 13:57 |
<godog> |
depool and reimage logstash1007 - T213898 |
[production] |
| 13:25 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 13:20 |
<gilles@deploy1001> |
Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 16s) |
[production] |
| 13:19 |
<gilles@deploy1001> |
Started deploy [3d2png/deploy@ca39432]: (no justification provided) |
[production] |
| 13:19 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) |
[production] |
| 13:19 |
<jbond42> |
restarting hhvm and updateing apache on deploy1001.eqiad.wmnet |
[production] |
| 13:18 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-upgrade |
[production] |
| 13:18 |
<gehel> |
restarting rolling upgrade on elasticsearch / cirrus / codfw - T215931 |
[production] |