ES索引瘦身 压缩——_source _all 均disable filed store为no,引入第三方DB存储原始数据,去掉pos倒排和doc_values,强制定期merge segments,将所有fileds合并为一个field big string
原始数据:835MB
ES 设置了_source _all disabled 且设置了仅仅存docs倒排
Wed Feb 22 11:58:27 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
435 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 11:59:43 CST 2017
done
不做任何压缩:
Wed Feb 22 14:56:04 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1656 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 14:57:54 CST 2017
done
curl -XPOST 'localhost:9200/hec_test3/_forcemerge'
bone@PEK1000074003:~/test_data/splunk_es_cmp$ du -sm ~/elasticsearch-2.4.1/data/
969 /home/bone/elasticsearch-2.4.1/data/
_all disabled:
Wed Feb 22 15:22:09 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1576 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:23:25 CST 2017
done
merge后:
787
_source disabled: field store为false 搜索不到东西了 如果不存DB的话
Wed Feb 22 15:26:06 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1288 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:27:45 CST 2017
done
merge后:
512
_source disabled: field store为yes
Wed Feb 22 15:33:28 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1583 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:34:45 CST 2017
done
merge后:
780
最终设置:
curl -XPUT 'http://localhost:9200/hec_test3' -d '
{
"mappings": {
"hec_type3": {
"_source": {
"enabled": false
},
"_all": {
"enabled": false
},
"properties": {
"event": {
"type": "string",
"include_in_all": false,
"doc_values": false,
"index_options": "docs"
}
}
}
}
}
'
压缩配置:
index.codec: best_compression
bash test_es24.sh
Wed Feb 22 15:39:39 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
892 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:40:36 CST 2017
done
merge后:
202 MB
50个字段拆分后:
用上述最终设置的效果:
Wed Feb 22 16:02:59 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1423 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:05:04 CST 2017
done
merge后:686
不做任何配置:
Wed Feb 22 16:08:30 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
2641 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:11:54 CST 2017
done
merge后:
1650
相关文章
- 【腾讯云ES】基于ES的游戏社区搜索服务实践
- OpenGL ES 着色语言
- 【ES三周年】初识ES的特点以及应用场景
- 【ES三周年】+Elasticsearch 在大数据的应用
- 【ES三周年】聊一聊关于Elasticsearch的那些事
- 【ES三周年】- Elasticsearch简介与使用环境安装
- 【ES三周年】SpringBoot整合elasticsearch存储抖音短视频数据实践
- IOS – OpenGL ES 图像色调分离噪点效果 GPUImagePosterizeFilter
- 【ES三周年】线下es环境数据迁移至腾讯云ES实例实践
- 库 利用ES搜索MySQL数据库的实现方法(es搜索mysql数据)
- 电商搜索ES与Redis的黄金组合(电商搜索es和redis)
- 使用ES查询Oracle新方法进行数据挖掘(es查询oracle)
- 使用ES数据导入Oracle让数据更加有序(es数据导入oracle)
- ES和Oracle一种新的数据处理方案(es和oracle)
- 使用Redis导入ES实现高效数据迁移(把redis数据导入es)