org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
2023-09-11 14:14:34 时间
flume在抽取MySQL数据到kafka时报错,如下
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96) at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:161) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: Failed to publish events at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:264) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96) at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:161) ... 3 more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可
a1.channels.ch-1.capacity = 10000
注意:Flume配置每行后边不能加注释,否则会导致配置不生效
相关文章
- EXCEL最大行数问题:org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1700)
- 使用命令进行Apache Kafka操作
- [Algorithm] 234. Palindrome Linked List / Reverse linked list
- 解决 Missing artifact org.apache.mina:mina-core:bundle:2.0.9
- CVE-2021-44228-Apache-Log4j-Rce漏洞反弹win&linux
- [Linux] Apache Gateway Timeout
- Apache Shiro的运行流程和权限控制方式分析
- 设置apache https服务
- 【异常】Maven构建出错,Could not transfer artifact xxx from/to central (https://repo.maven.apache.org/maven2)
- Apache ServiceComb 开源两周年,聊聊其与微服务的前世今生
- apache commons-pool2中GenericObjectPoolConfig的maxTotal、maxIdle、minIdle属性理解
- 为什么Nginx性能比Apache高
- 【Apache Spark 】第 9 章使用 Apache Spark构建可靠的数据湖
- 轻量级压测工具Apache Bench实战