【ELK】2 Logstash安裝
一般模式 FileBeat(更輕量化) --> Logstash
1、Logstash是什麼?
Logstash是一個日誌收集、過濾、轉發的中介軟體,主要負責將各條業務線的各類日誌統一收集、過濾後,轉發給 Elasticsearch 進行下一步處理
2、安裝Logstash
logstash無需安裝,直接解壓就行。收集日誌需要寫符合業務需求的配置檔案。下面從最基本的控制檯輸入輸出開始,一步步配置符合內心期望的收集、過濾等配置。
1)簡單安裝
# cd /usr/local/src
# tar -zxvf logstash-5.2.2.tar.gz
# mv logstash-5.2.2 /usr/local/logstash
2)為logstash安裝x-pack
# cd /usr/local/logstash/bin
# ./logstash-plugin install x-pack
[[email protected] bin]# ./logstash-plugin install x-pack
Downloading file: https://artifacts.elastic.co/downloads/logstash-plugins/x-pack/x-pack-5.2.2.zip
Downloading [=============================================================] 100%
Installing file: /tmp/studtmp-e97bf0800211a6de985aa19c225dd2a5dce0d369da9f5eb975df52c0adee/x-pack-5.2.2.zip
Install successful
[[email protected] bin]#
3)建立配置檔案,指定資料流向,在目錄下建立一個simple.conf,內容如下
input { beats { port => "5044" } } # 資料過濾 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } geoip { source => "clientip" } } # 輸出到本機的 ES output { elasticsearch { hosts => [ "127.0.0.1:9200" ] } }
3)上述配置輸出日誌到ES,啟動命令如下:
# cd /usr/local/logstash/bin
# ./logstash -f /usr/local/logstash/config/simple.conf --config.reload.automatic
啟動時報錯:LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://127.0.0.1:9200/'"}
原因:之前給ES 安裝了x-pack外掛,訪問需要登入賬號和密碼
解決:調整logstash配置檔案,在output中 訪問es時增加 user、password
# cd /usr/local/logstash/config
# vi + simple.conf output { elasticsearch { hosts => [ "127.0.0.1:9200" ]
user => elastic
password=>changeme } }
# cd /usr/local/logstash/bin
# ./logstash -f /usr/local/logstash/config/simple.conf --config.reload.automatic 重新啟動
...
[2018-08-03T19:08:53,852][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb"}
[2018-08-03T19:08:53,879][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-08-03T19:08:54,307][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-08-03T19:08:54,343][INFO ][logstash.pipeline ] Pipeline main started
[2018-08-03T19:08:54,408][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
參考地址https://blog.csdn.net/Ahri_J/article/details/79609444
3、安裝FileBeats
下載地址: https://www.elastic.co/downloads/beats/filebeat
安裝檔案:filebeat-5.2.2-linux-x86_64.tar.gz
1)簡單安裝FileBeats
# cd /usr/local/src
# tar -zxvf filebeat-5.2.2-linux-x86_64.tar.gz
# mv filebeat-5.2.2-linux-x86_64 /usr/local/filebeat
2) 配置 filebeats.yml ,指定讀取日誌目錄,指定輸出到本機logstash
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /home/daxiang/logs/mobile/*.log
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
3)啟動
測試啟動命令
# ./filebeat -e -c filebeat.yml -d "publish"
正常後臺執行啟動,日誌開始採集
# nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &
4、Logstash基本學習
1)執行最基本的logstash管道
# cd /usr/local/logstash/bin
# ./logstash -e 'input { stdin {} } output { stdout {}}'
執行效果如下
[[email protected] bin]# ./logstash -e 'input { stdin {} } output { stdout {}}'
Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2018-07-25T16:18:00,220][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/logstash/data/queue"}
[2018-07-25T16:18:00,238][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"1850a7c6-e6b8-43ae-9079-6e8d27be4173", :path=>"/usr/local/logstash/data/uuid"}
[2018-07-25T16:18:00,410][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-07-25T16:18:00,433][INFO ][logstash.pipeline ] Pipeline main started
The stdin plugin is now waiting for input:
[2018-07-25T16:18:00,506][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
hello (在控制檯輸入hello,可看到如下輸出)
2018-07-25T08:18:34.363Z test hello
hello world
2018-07-25T08:19:15.051Z test hello world (當前輸出格式 時間戳 主機名 輸入內容)
2)配置輸出時,指定使用codec
建立一個logstash-simple.conf
# cd /usr/local/
# touch logstash-simple.conf
# vi logstash-simple.conf
#輸入來源
input { stdin {} }
#指定輸出格式
output {
stdout{
codec=>rubydebug
}
}
# cd /usr/local/logstash/bin
# ./logstash -f /usr/local/logstash/config/logstash-simple.conf
執行結果如下
[[email protected] bin]# ./logstash -f /usr/local/logstash/config/logstash-simple.conf
Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2018-07-25T16:53:33,755][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2018-07-25T16:53:33,794][INFO ][logstash.pipeline ] Pipeline main started
The stdin plugin is now waiting for input:
[2018-07-25T16:53:33,849][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
hello 輸入hello,返回了JSON
{
"@timestamp" => 2018-07-25T08:53:45.928Z,
"@version" => "1",
"host" => "test",
"message" => "hello"
}
5、Logstash配置指定日誌的切割正則,需用到ruby做正則,待後續完善。