电商数仓3.0 数据采集模块之采集日志Flume安装
发布时间
阅读量：
阅读量
资料
文档访问路径：http://flume.apache.org/FlumeUserGuide.html
官方网站链接：http://flume.apache.org/
获取来源：http://archive.apache.org/dist/flume/
技术博客页面：<>
下载链接：https://pan.baidu.com/s/1EWOiiNWyIWTYjxYfBfZplw
请关注提取码：3hti
集群规划
删除guava-11.0.2.jar
将lib文件夹下的guava-11.0.2.jar删除以兼容Hadoop 3.1.3
[scorpion@warehouse102 module]$ rm /opt/module/flume-1.9.0/lib/guava-11.0.2.jar
配置flume-env.sh文件
[scorpion@warehouse102 conf]$ mv flume-env.sh.template flume-env.sh
[scorpion@warehouse102 conf]$ vim flume-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144
编写file-flume-kafka.conf
// 在 /opt/module/flume-1.9.0/conf 目录下创建file-flume-kafka.conf文件
[scorpion@warehouse102 conf]$ vim file-flume-kafka.conf
#为各组件命名
a1.sources = r1
a1.channels = c1
#描述source
a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/applog/log/app.*
a1.sources.r1.positionFile = /opt/module/flume-1.9.0/taildir_position.json
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = com.warehouse.flume.interceptor.ETLInterceptor$Builder
#描述channel
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = warehouse102:9092,warehouse103:9092
a1.channels.c1.kafka.topic = topic_log
a1.channels.c1.parseAsFlumeEvent = false
#绑定source和channel以及sink和channel的关系
a1.sources.r1.channels = c1
// com.warehouse.flume.interceptor.ETLInterceptor$Builder 是自定义的拦截器的全类名
编写ETLInterceptor
仓库：https://github.com/SmallScorpion/flume-interceptor
将带有依赖项的JAR文件放置到 warehouse102 的 /opt/module/flume-1.9.0/lib 目录下
分发并启动
// 分发
[scorpion@warehouse102 module]$ xsync flume-1.9.0/
// 在102 和 103 分别启动Flume
[scorpion@warehouse102 flume-1.9.0]$ bin/flume-ng agent --name a1 --conf-file conf/file-flume-kafka.conf &
[scorpion@warehouse103 flume-1.9.0]$ bin/flume-ng agent --name a1 --conf-file conf/file-flume-kafka.conf &日志采集Flume群起脚本
[scorpion@warehouse102 bin]$ vim flume-collection-log.sh
#! /bin/bash
# flume 日志采集
case $1 in
"start"){
for i in warehouse102 warehouse103
do
echo " --------启动 $i 采集flume-------"
ssh $i "nohup/opt/module/flume-1.9.0/bin/flume-ng agent --conf-file /opt/module/flume-1.9.0/conf/file-flume-kafka.conf --name a1 -Dflume.root.logger=INFO,LOGFILE >/opt/module/flume-1.9.0/log1.txt 2>&1 &"
done
};;
"stop"){
for i in warehouse102 warehouse103
do
echo " --------停止 $i 采集flume-------"
ssh $i "ps -ef | grep file-flume-kafka | grep -v grep |awk '{print \$2}' | xargs -n1 kill -9 "
done
};;
esac
// 增加权限
[scorpion@warehouse102 bin]$ chmod 777 flume-collection-log.sh全部评论 (0)
还没有任何评论哟~