Flume 读取JMS 消息队列消息,并将消息写入HDFS
利用Apache Flume 读取JMS 消息队列消息。并将消息写入HDFS,flume agent配置例如以下:
flume-agent.conf
#name the components on this agent
agentHdfs.sources = jms_source
agentHdfs.sinks = hdfs_sink
agentHdfs.channels = mem_channel
# Describe/configure the source
agentHdfs.sources.jms_source.type = jms
# Bind to all interfaces
agentHdfs.sources.jms_source.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
agentHdfs.sources.jms_source.connectionFactory = ConnectionFactory
agentHdfs.sources.jms_source.destinationName = BUSINESS_DATA #AMQ queue
agentHdfs.sources.jms_source.providerURL = tcp://hadoop-master:61616
agentHdfs.sources.jms_source.destinationType = QUEUE
# Describe the sink
agentHdfs.sinks.hdfs_sink.type = hdfs
agentHdfs.sinks.hdfs_sink.hdfs.path hdfs://hadoop-master/data/flume/%Y-%m-%d/%H
agentHdfs.sinks.hdfs_sink.hdfs.filePrefix = %{hostname}/events-
agentHdfs.sinks.hdfs_sink.hdfs.maxOpenFiles = 5000
agentHdfs.sinks.hdfs_sink.hdfs.batchSize= 500
agentHdfs.sinks.hdfs_sink.hdfs.fileType = DataStream
agentHdfs.sinks.hdfs_sink.hdfs.writeFormat =Text
agentHdfs.sinks.hdfs_sink.hdfs.rollSize = 0
agentHdfs.sinks.hdfs_sink.hdfs.rollCount = 1000000
agentHdfs.sinks.hdfs_sink.hdfs.rollInterval = 600
agentHdfs.sinks.hdfs_sink.hdfs.useLocalTimeStamp = true
# Use a channel which buffers events in memory
agentHdfs.channels.mem_channel.type = memory
agentHdfs.channels.mem_channel.capacity = 1000
agentHdfs.channels.mem_channel.transactionCapacity = 100
# Bind the source and sink to the channel
agentHdfs.sources.jms_source.channels = mem_channel
agentHdfs.sinks.hdfs_sink.channel = mem_channel
相关文章
- Python微信公众号教程基础篇——收发文本消息
- 第二百九十一节,RabbitMQ多设备消息队列-安装与简介
- JavaScript 消息框,警告框,确认框,提示框
- RabbitMQ消息队列(四):分发到多Consumer(Publish/Subscribe)
- Sparkstreaming读取Kafka消息再结合SparkSQL,将结果保存到HBase
- Spring整合ActiveMQ完成消息队列MQ编程
- Windows下使用python库 curses遇到错误消息的解决方案
- IIS 编译器错误消息: CS0016未能写入输出文件“c:WINDOWSMicrosoft.NETFrameworkv1.1.4322Temporary ASP.NET Filesroo
- [手游新项目历程]第12天-利用共享内存实现消息队列
- SpringBoot整合ActiveMQ消息组件
- y144.第八章 Servless和Knative从入门到精通 -- 消息系统基础(八)
- 动态方法决议 和 消息转发
- rabbitmq 延时队列 插件方式实现 每条消息都延时自己时间
- 基于Python语言使用RabbitMQ消息队列(四)
- 异步通信rabbitmq——消息重试
- 消息队列mq总结(重点看,比较了主流消息队列框架)
- RocketMQ消息队列——消息存储详解
- 消息队列:解耦、异步、削峰,现有MQ对比以及新手入门该如何选择MQ?