ClickHouse系列--Too many partitions for single INSERT block (more than 100)
2023-04-18 12:43:14 时间
批量插入时,这一批数据是按照天分区的,一批数据里的日期跨度为一年,也就是单次插入可能产生365个分区,导致后台异步合并数据出现问题。 解决方案:单批次写入数据,要控制分区数量。
exception, code: 252, host: 10.58.10.243, port: 8123; Code: 252, e.displayText() = DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). (version 21.5.6.6 (official build))
ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 252, host: 10.58.10.243, port: 8123; Code: 252, e.displayText() = DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). (version 21.5.6.6 (official build))
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:58) ~[clickhouse-jdbc-0.2.4.jar:na]
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:28) ~[clickhouse-jdbc-0.2.4.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(ClickHouseStatementImpl.java:875) ~[clickhouse-jdbc-0.2.4.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:851) ~[clickhouse-jdbc-0.2.4.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:824) ~[clickhouse-jdbc-0.2.4.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:817) ~[clickhouse-jdbc-0.2.4.jar:na]
at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:335) ~[clickhouse-jdbc-0.2.4.jar:na]
at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:320) ~[clickhouse-jdbc-0.2.4.jar:na]
at cn.java4all.core.consumer.ClickHouseConsumer.onBulkMessage(ClickHouseConsumer.java:87) ~[classes/:na]
at cn.java4all.core.consumer.AbstractConsumer.doConsume(AbstractConsumer.java:69) [classes/:na]
at cn.fraudmetrix.module.kafka.consumer.ConcurrentPoller.lambda$handle$1(ConcurrentPoller.java:98) [module-kafka-3.2.3-20210708.074058-50.jar:na]
at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1626) ~[na:1.8.0_211]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[na:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_211]
Caused by: java.lang.Throwable: Code: 252, e.displayText() = DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). (version 21.5.6.6 (official build))
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:53) ~[clickhouse-jdbc-0.2.4.jar:na]
... 15 common frames omitted
相关文章
- HarmonyOS基础技术赋能之分布式数据服务功能
- HarmonyOS自定义控件之速度检测VelocityDetector
- HarmonyOS服务卡片开发知识总结
- Firefox 91 支持用户一键清除指定或全部网站的 Cookie
- HarmonyOS小游戏:吃豆豆---基于分布式数据库与分布式任务调度
- Windows 10/11 福利:微软为所有安卓手机带来 Windows 剪贴板同步功能
- 就很凸然,电脑里多出了一个文件···
- HarmonyOS 分布式亲子教育
- 十大面向Windows的免费数据恢复工具
- Windows 7还能升级Windows 11吗?微软披露Windows11升级细节
- 从模型复杂度角度来理解过拟合现象
- 拉格朗日乘数法,一种计算条件极值的方式
- elasticsearch高亮之highlight原理
- elasticsearch查询之大数据集分页查询
- elasticsearch之多索引查询
- 如何使用注解优雅的记录操作日志 | 萌新写开源 01
- elasticsearch支持大table格式数据的搜索
- 解析fiddler返回的部分数据。
- SpringBoot快速配置多数据源(整合MyBatis)
- WebSocket集群分布式改造:实现多人在线聊天室