Spark Streaming 项目实战(2)—— Flume 对接python日志产生器,和Kafka

1 Flume 对接日志产生器

1.1 Flume 配置文件

streaming_project.conf

exec-memory-logger.sources = exec-source
exec-memory-logger.sinks = logger-sink
exec-memory-logger.channels = memory-channel

exec-memory-logger.sources.exec-source.type = exec
exec-memory-logger.sources.exec-source.command = tail -F /home/hadoop/tempdata/access.log
exec-memory-logger.sources.exec-source.shell = /bin/sh -c

exec-memory-logger.channels.memory-channel.type = memory

exec-memory-logger.sinks.logger-sink.type = logger

exec-memory-logger.sources.exec-source.channels = memory-channel
exec-memory-logger.sinks.logger-sink.channel = memory-channel

1.2 启动 Flume Agent

flume-ng agent \
--name exec-memory-logger \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/myconf/streaming_project.conf \
-Dflume.root.logger=INFO,console

在这里插入图片描述

2 Flume 对接 Kafka

  • 首先启动 zookeeper ,zkServer.sh start
    在这里插入图片描述

  • 启动 Kafka Server
    kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties
    在这里插入图片描述

2.1 修改Flume配置文件,将数据 sink 到Kafka

streaming_project2.conf

exec-memory-kafka.sources = exec-source
exec-memory-kafka.sinks = kafka-sink
exec-memory-kafka.channels = memory-channel

exec-memory-kafka.sources.exec-source.type = exec
exec-memory-kafka.sources.exec-source.command = tail -F /home/hadoop/tempdata/access.log
exec-memory-kafka.sources.exec-source.shell = /bin/sh -c

exec-memory-kafka.channels.memory-channel.type = memory

exec-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
exec-memory-kafka.sinks.kafka-sink.brokerList = node1:9092
exec-memory-kafka.sinks.kafka-sink.topic = streamingtopic
exec-memory-kafka.sinks.kafka-sink.batchSize = 5
exec-memory-kafka.sinks.kafka-sink.requiredAcks = 1

exec-memory-kafka.sources.exec-source.channels = memory-channel
exec-memory-kafka.sinks.kafka-sink.channel = memory-channel

2.2 启动kafka消费者

kafka-console-consumer.sh --zookeeper node1:2181 --topic streamingtopic
  • 启动 FLume
flume-ng agent \
--name exec-memory-kafka \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/myconf/streaming_project2.conf \
-Dflume.root.logger=INFO,console

在这里插入图片描述

然后观察消费者的窗口
在这里插入图片描述

已标记关键词 清除标记
©️2020 CSDN 皮肤主题: 书香水墨 设计师:CSDN官方博客 返回首页