日志采集框架Flume的安裝及使用

2019-11-08 00:14:51

字體：大中小

來源：轉載

供稿：網友

日志采集框架Flume的安裝及使用

1.Flume介紹

1.1.Flume概述

Flume是一個分布式、可靠、和高可用(舊版Flume og才有高可用)的海量日志采集、傳輸和聚合的系統。Flume可以采集文件，socket數據包等各種形式源數據，又可以將采集到的數據輸出到HDFS、hbase、hive、kafka等眾多外部存儲系統中一般的采集需求，通過對flume的簡單配置即可實現Flume針對特殊場景也具備良好的自定義擴展能力，因此，flume可以適用于大部分的日常數據采集場景

1.2.運行機制

1、Flume分布式系統中最核心的角色是agent，flume采集系統就是由一個個agent所連接起來形成2、每一個agent相當于一個數據傳遞員，內部有三個組件：a)Source：采集源，用于跟數據源對接，以獲取數據b)Sink：下沉地，采集數據的傳送目的，用于往下一級agent傳遞數據或者往最終存儲系統傳遞數據c)Channel：angent內部的數據傳輸通道，用于從source中將數據傳遞到sink

Markdown

1.3.Flume采集系統結構圖

1.3.1.簡單結構

單個agent采集數據

Markdown

1.3.2.復雜結構

多級agent之間串聯

Markdown

2.安裝Flume

2.1.解壓Flume壓縮文件到指定目錄

[root@node02 software]# tar -zxvf apache-flume-1.6.0-bin.tar.gz -C /opt/modules/

2.2.文件重命名

[root@node02 modules]# mv apache-flume-1.6.0-bin flume-1.6.0You have new mail in /var/spool/mail/root[root@node02 modules]# lltotal 24drwxr-xr-x. 9 matrix matrix 4096 Jan 7 13:44 elasticsearch-2.4.2drwxr-xr-x. 7 root root 4096 Jan 24 13:09 flume-1.6.0drwxr-xr-x. 12 matrix matrix 4096 Jan 23 21:00 hadoop-2.5.1drwxr-xr-x. 8 root root 4096 Jan 23 18:43 hive-1.2.1drwxr-xr-x. 3 matrix matrix 4096 Dec 19 16:01 journalnodedrwxr-xr-x. 12 matrix matrix 4096 Dec 17 21:20 zookeeper

2.3.配置Flume環境變量

[root@node02 ~]# ls -a. anaconda-ks.cfg .bash_logout .bashrc .hivehistory install.log.syslog .MySQL_history .ssh zookeeper.out.. .bash_history .bash_PRofile .cshrc install.log jdk-7u79-linux-x64.rpm .pki .tcshrc[root@node02 ~]# vi .bash_profile

Markdown

export FLUME_HOME=/opt/modules/flume-1.6.0export PATH=$PATH:$FLUME_HOME/bin

Markdown

2.4.使配置生效

[root@node02 ~]# source .bash_profile

2.5.采集文件到HDFS

采集需求：比如業務系統使用log4j生成的日志，日志內容不斷增加，需要把追加到日志文件中的數據實時采集到hdfs根據需求，首先定義以下3大要素采集源，即source——監控文件內容更新：exec 'tail -F file'下沉目標，即sink——HDFS文件系統：hdfs sinkSource和sink之間的傳遞通道——channel，可用file channel也可以用內存channel

2.5.1.配置Flume配置文件

[root@node02 flume-1.6.0]# vi conf/tail-hdfs.conf

Markdown

# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1#exec 指的是命令# Describe/configure the sourcea1.sources.r1.type = exec#F根據文件名追中, f根據文件的nodeid追中a1.sources.r1.command = tail -F /home/hadoop/log/test.loga1.sources.r1.channels = c1# Describe the sink#下沉目標a1.sinks.k1.type = hdfsa1.sinks.k1.channel = c1#指定目錄, flum幫做目的替換a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/#文件的命名, 前綴a1.sinks.k1.hdfs.filePrefix = events-#10 分鐘就改目錄a1.sinks.k1.hdfs.round = truea1.sinks.k1.hdfs.roundValue = 10a1.sinks.k1.hdfs.roundUnit = minute#文件滾動之前的等待時間(秒)a1.sinks.k1.hdfs.rollInterval = 3#文件滾動的大小限制(bytes)a1.sinks.k1.hdfs.rollSize = 500#寫入多少個event數據后滾動文件(事件個數)a1.sinks.k1.hdfs.rollCount = 20#5個事件就往里面寫入a1.sinks.k1.hdfs.batchSize = 5#用本地時間格式化目錄a1.sinks.k1.hdfs.useLocalTimeStamp = true#下沉后, 生成的文件類型，默認是Sequencefile，可用DataStream，則為普通文本a1.sinks.k1.hdfs.fileType = DataStream# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1

Markdown

2.6.寫數據到指定的文件中

[root@node02 flume-1.6.0]# mkdir -p /home/hadoop/log[root@node02 flume-1.6.0]# touch /home/hadoop/log/test.log[root@node02 ~]# while true> do> echo 11111111111 >> /home/hadoop/log/test.log> sleep 0.6> donewhile truedoecho 11111111111 >> /home/hadoop/log/test.logsleep 0.6done

Markdown

[root@node02 flume-1.6.0]# tail -f /home/hadoop/log/test.log

2.7.啟動Flume日志收集

注意：檢查Hadoop HDFS是否啟動，如沒有啟動，則啟動[root@node02 flume-1.6.0]# ./bin/flume-ng agent -c conf -f conf/tail-hdfs.conf -n a1

Markdown

2.8.通過Hadoop Web UI查看Flume在HDFS上創建的目錄

[root@node02 hadoop-2.5.1]# ./bin/hdfs dfs -ls -R /flumedrwxr-xr-x - root supergroup 0 2017-01-24 13:36 /flume/eventsdrwxr-xr-x - root supergroup 0 2017-01-24 13:36 /flume/events/17-01-24drwxr-xr-x - root supergroup 0 2017-01-24 13:38 /flume/events/17-01-24/1330-rw-r--r-- 3 root supergroup 140 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236169660-rw-r--r-- 3 root supergroup 140 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236169661-rw-r--r-- 3 root supergroup 70 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236169662-rw-r--r-- 3 root supergroup 77 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236189653-rw-r--r-- 3 root supergroup 77 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236195683-rw-r--r-- 3 root supergroup 77 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236201641-rw-r--r-- 3 root supergroup 77 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236207790-rw-r--r-- 3 root supergroup 84 2017-01-24 13:36 /flume/events/17-01-24/1330/events-.1485236213809-rw-r--r-- 3 root supergroup 70 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236219808-rw-r--r-- 3 root supergroup 84 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236225867-rw-r--r-- 3 root supergroup 77 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236231852-rw-r--r-- 3 root supergroup 70 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236238116-rw-r--r-- 3 root supergroup 84 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236244133-rw-r--r-- 3 root supergroup 63 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236250160-rw-r--r-- 3 root supergroup 56 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236254744-rw-r--r-- 3 root supergroup 42 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236260456-rw-r--r-- 3 root supergroup 35 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236264210-rw-r--r-- 3 root supergroup 35 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236267832-rw-r--r-- 3 root supergroup 49 2017-01-24 13:37 /flume/events/17-01-24/1330/events-.1485236271410-rw-r--r-- 3 root supergroup 84 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236275630-rw-r--r-- 3 root supergroup 77 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236281581-rw-r--r-- 3 root supergroup 77 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236287587-rw-r--r-- 3 root supergroup 84 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236293646-rw-r--r-- 3 root supergroup 70 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236299642-rw-r--r-- 3 root supergroup 49 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236305888-rw-r--r-- 3 root supergroup 70 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236311177-rw-r--r-- 3 root supergroup 35 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236315362-rw-r--r-- 3 root supergroup 77 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236320019-rw-r--r-- 3 root supergroup 77 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236324629-rw-r--r-- 3 root supergroup 42 2017-01-24 13:38 /flume/events/17-01-24/1330/events-.1485236330636

Markdown

3.Flume多個agent連接

Markdown

從tail命令獲取數據發送到avro端口(tail->avro)另一個節點可配置一個avro源來中繼數據，發送外部存儲(avro->log)

3.1.在node03上安裝Flume

3.2.配置node02上的Flume

從tail命令獲取數據發送到avro端口[root@node02 flume-1.6.0]# vi conf/tail-avro.conf# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.command = tail -F /home/hadoop/log/test.loga1.sources.r1.channels = c1# Describe the sink#綁定的不是本機, 是另外一臺機器的服務地址, sink端的avro是一個發送端, avro的客戶端, 往node03這個機器上發a1.sinks = k1a1.sinks.k1.type = avroa1.sinks.k1.channel = c1a1.sinks.k1.hostname = 192.168.230.12a1.sinks.k1.port = 4141a1.sinks.k1.batch-size = 2# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1

3.2.1.運行node02上的Flume,發送數據給node03

[root@node02 flume-1.6.0]# ./bin/flume-ng agent -c conf -f conf/tail-avro.conf -n a1

Markdown

3.3.配置node03上的Flume

配置一個avro源來中繼數據，發送外部存儲/hdfs(avro->log)[root@node03 flume-1.6.0]# vi conf/avro-hdfs.conf# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the source#source中的avro組件是接收者服務, 綁定本機a1.sources.r1.type = avroa1.sources.r1.channels = c1a1.sources.r1.bind = 0.0.0.0a1.sources.r1.port = 4141# Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1