2.1 輸入插件在"hello World" 示例中,我們已經見到并介紹了Logstash 的運行流程和配置的基礎語法。請記住一個原則: Logstash 配置一定要有一個input和一個output在演示過程中,如果沒有寫明input,默認就會使用 logstash-input-stdin同理,沒有寫明的output 就是logstash-output-stdout2.1.1 標準輸入[elk@Vsftp logstash]$ cat stdin.conf input {  stdin {  add_field =>{"key11"=>"value22"}  codec=>"plain"  tags=>["add"]  type=>"std" }}output {     stdout {                        codec => rubydebug                } }[elk@Vsftp logstash]$ logstash -f stdin.conf Settings: Default pipeline workers: 4Pipeline main startedabc123{       "message" => "abc123",      "@version" => "1",    "@timestamp" => "2017-02-08T02:14:53.476Z",          "type" => "std",         "key11" => "value22",          "tags" => [        [0] "add"    ],          "host" => "Vsftp"}[elk@Vsftp logstash]$ cat stdin.conf input {  stdin {  add_field =>{"key11"=>"value22"}  codec=>"plain"  tags=>["add","xxyy"]  type=>"std" }}output {     stdout {                        codec => rubydebug                } }[elk@Vsftp logstash]$ logstash -f stdin.conf Settings: Default pipeline workers: 4Pipeline main startedthis is scan{       "message" => "this is scan",      "@version" => "1",    "@timestamp" => "2017-02-08T02:15:39.183Z",          "type" => "std",         "key11" => "value22",          "tags" => [        [0] "add",        [1] "xxyy"    ],          "host" => "Vsftp"}根據tags 判斷:[elk@Vsftp logstash]$ cat stdin.conf input {  stdin {  add_field =>{"key11"=>"value22"}  codec=>"plain"  tags=>["add","xxyy"]  type=>"std" }}output {  if "tttt" in [tags]{   stdout {    codec=>rubydebug{}     }  }   else if "add" in [tags]{   stdout {    codec=>json     }  }    }[elk@Vsftp logstash]$ logstash -f stdin.conf Settings: Default pipeline workers: 4Pipeline main startedyyyyyjjjj{"message":"yyyyyjjjj","@version":"1","@timestamp":"2017-02-08T02:20:42.833Z","type":"std","key11":"value22","tags":["add","xxyy"],"host":"Vsftp"}2.1.2  文件輸入:logstash 使用一個名叫FileWatch的Ruby Gem庫來監聽文件變化。這個庫支持glob展開文件路徑,而且會記錄一個叫.sincedb的數據庫文件來跟蹤被監聽日志文件的當前讀取位置[elk@Vsftp logstash]$ cat log.conf input {  file {   path =>["/var/log/*.log","/var/log/mm"]   type=>"system"  start_position =>"beginning"}}output {        stdout {                        codec => rubydebug                } }[elk@Vsftp logstash]$ logstash -f log.conf Settings: Default pipeline workers: 4Pipeline main started{       "message" => "11111111111",      "@version" => "1",LogStash::Inputs::File 只是在進程運行的注冊階段初始化一個FileWatch對象。所以它不能支持類型fluentd 那樣的path=>"2.1.3 TCP 輸入 未來你可能會用Redis 服務器或者其他的消息隊列系統來作為Logstash Broker的角色。不過Logstash 其實也有自己的TCP/UDP 插件,在臨時任務的時候,也算能用,尤其是測試環境。[elk@Vsftp logstash]$ cat tcp.conf input { tcp {   port =>8888   mode=>"server"  ssl_enable =>false }}output {        stdout {                        codec => rubydebug                }}[elk@Vsftp logstash]$ logstash -f tcp.conf Settings: Default pipeline workers: 4Pipeline main started{       "message" => "9999999999",      "@version" => "1",    "@timestamp" => "2017-02-08T03:02:43.746Z",          "host" => "127.0.0.1",          "port" => 47187}{       "message" => "000000000",      "@version" => "1",    "@timestamp" => "2017-02-08T03:02:43.747Z",          "host" => "127.0.0.1",          "port" => 47187}Vsftp:/var/log#  nc 127.0.0.1 8888 < mmVsftp:/var/log# cat mm99999999990000000002.1.4 syslog輸入: syslog 可能是運維領域最流行的數據傳輸協議了,當你想從設備上收集系統日志的時候,syslog 應該會是你第一選擇。尤其是網絡設備介紹 如何把Logstash 配置成一個syslog 服務器來接收數據。2.2 編解碼配置:Codec 是Logstash 從1.3.0 開始引入的概念(Codec 來自Coder/decoder 兩個單詞的首字母縮寫)我們在第一個"Hello World" 用例就已經用過Codec了 rubydebug就是一種Codec 雖然它一般只會在stdout 插件中,作為配置測試或者調試的工具2.2.1 JSON 編解碼:2.2.2  多行事件編碼有些時候,應用程序調試日志會包含非常豐富的內容,為一個事件打印出很多行內容。這種日志通常都很難通過命令行解析的方式做分析而Logstash 正為此準備好了 codec/multiline 插件！當然,multiline插件也可以用于其他類似的堆棧信息Vsftp:/home/elk/logstash# cat multi.conf input {  stdin {  codec=>multiline {  pattern =>"^/["  negate =>true what=>"PRevious"  }}}output {        stdout {                        codec => rubydebug                }}[elk@Vsftp logstash]$ logstash -f multi.conf Settings: Default pipeline workers: 4Pipeline main started[Aug/02/03 11:45:27] aaaabbbbcccc[Aug/02/03 11:45:27]  998877{    "@timestamp" => "2017-02-08T05:27:07.442Z",       "message" => "[Aug/02/03 11:45:27] aaaa/nbbbb/ncccc",      "@version" => "1",          "tags" => [        [0] "multiline"    ],          "host" => "Vsftp"}其實這個插件的原理很簡單,就是把當前行的數據添加到前面一行后面,直到新進的當前行匹配^/[正則為止。2.3 過濾器配置:2.3.1 date時間處理之前章節已經提過,logstash-filter-date插件可以用來轉換你的日志記錄中的時間字符串,變成LogStash::Timestamp 對象,然后轉存到@timestamp 字段里因為在稍后的logstash-outout-elasticsearch 中常用的%{+YYYY.MM.dd}這種寫法必須讀取@timestamp 數據%{TIMESTAMP_ISO8601:time}  匹配如下時間格式:2011-04-18 08:20:112011-04-18 08:20:11,108[elk@Vsftp logstash]$ logstash -f stdin.conf Settings: Default pipeline workers: 4Pipeline main startedaaaaabbbbbccccc{"message":"aaaaabbbbbccccc","@version":"1","@timestamp":"2017-02-08T05:44:44.165Z","type":"std","key11":"value22","tags":["add","xxyy"],"host":"Vsftp"}