es 5.x 版本使用srcoll scan解决数据太大,分页报错的问题
2023-09-27 14:22:12 时间
@Override
public Page<RealTimeMonitorDomain> querySimlarInfoList(int pageSize, int page, String groupId,String id) {
//1.初始化条件
NativeSearchQueryBuilder nsb = new NativeSearchQueryBuilder();//总的查询条件
BoolQueryBuilder bools = QueryBuilders.boolQuery();
//2.设置查询
bools.must(QueryBuilders.termQuery("groupId",groupId));
bools.mustNot(QueryBuilders.termQuery("id",id));
nsb.withQuery(bools);
//3.设置排序
nsb.withSort(new FieldSortBuilder("pubDate").order(SortOrder.DESC));
//4.设置分页
nsb.withPageable(new PageRequest(page-1, pageSize));//pagenum是从0开始,坑爹啊!
//5.查询的建立
SearchQuery searchQuery = nsb.build();//查询建立
logger.info("相似度列表查询sql:"+searchQuery.getQuery().toString());
// Page<RealTimeMonitorDomain> pageable = elasticsearchTemplate.queryForPage(searchQuery,RealTimeMonitorDomain.class);
Paging paging=new Paging();
List<RealTimeMonitorDomain> rtmdList=new ArrayList<RealTimeMonitorDomain>();
String scrollId = elasticsearchTemplate.scan(searchQuery,1000,false);
boolean hasRecords = true;
while (hasRecords) {
Page<RealTimeMonitorDomain> singlePage = elasticsearchTemplate.scroll(scrollId, 5000L, new SearchResultMapper() {
@Override
public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
List<RealTimeMonitorDomain> chunk = new ArrayList<RealTimeMonitorDomain>();
for (SearchHit searchHit : response.getHits()) {
if (response.getHits().getHits().length <= 0) {
return null;
}
RealTimeMonitorDomain user = new RealTimeMonitorDomain();
user.setId(searchHit.getId());
user.setTitle(searchHit.getSource().get("message").toString());
chunk.add(user);
}
if (chunk.size() > 0) {
return new AggregatedPageImpl<T>((List<T>) chunk);
}
return null;
}
});
if (singlePage!= null) {
rtmdList.addAll(singlePage.getContent());
} else {
hasRecords = false;
}
paging.setTotalCount(singlePage.getTotalElements());
paging.setTotalPage(singlePage.getTotalPages());
}
elasticsearchTemplate.clearScroll(scrollId);
paging.setPage(page);
paging.setPageSize(pageSize);
paging.setResult(rtmdList);
return pageable;
}
相关文章
- ES 终于可以搜到”悟空哥“了!
- ElasticSearch from + size 分页查询过程分析,及其官方ES深度分页性能优化方法
- ES PS TS 流的区别
- elasticsearch系列一elasticsearch(ES简介、安装&配置、集成Ikanalyzer)
- 从es中拉取全部数据/大量数据 使用scroll+scan避免深分页
- Android下Opengl ES实现单屏幕双眼显示
- ES Aggs count distinct group by聚合排序查询
- Elasticsearch教程(6) ES桶聚合Query DSL-Terms Aggregation
- 【笔记】再学JavaScript ES(6-10)全版本语法——反射机制及应用
- OpenGL ES案例-抖音系滤镜实现
- 推荐一款强大的 Android OpenGL ES 调试工具
- OpenGL ES 多目标渲染(MRT)
- elasticsearch系列七:ES Java客户端-Elasticsearch Java client(ES Client 简介、Java REST Client、Java Client、Spring Data Elasticsearch)
- java实现es的search after查询(三种方式详解)
- ES 基本操作
- React Conf 2017 干货总结 1: React + ES next = ♥
- (原)关于OpenSL ES播放音频数据的一个奇怪的问题