通過logstash-input-mongodb外掛將mongodb資料匯入ElasticSearch
安裝:logstash-input-mongodb外掛
環境:ElasticSearch5.5.2、logstash5.5.2
步驟:
1.進入logstash 下bin目錄 檢視已安裝的外掛:
./logstash-plugin list
2.沒有logstash-input-mongodb外掛那麼:
./logstash-plugin install logstash-input-mongodb
3.安裝失敗:換成國內映象
4.沒有gem命令的先安裝:
yum install gem
5.可以先看下映象庫地址命令如下:
gem sources -l
可以看到地址是:https://rubygems.org/
6.現在替換為國內的ruby-china庫:
gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/
#在檢視
gem sources -l
7.此時一切換成功,當然,並沒有完成,需要進入logstash目錄對 Gemfile檔案 進行編輯:
vim Gemfile
將檔案裡的 source "https://rubygems.org" 換成 source "https://gems.ruby-china.org"
8.wq儲存退出,好了進入bin再執行:
./logstash-plugin install logstash-input-mongodb
9.依然失敗的話:
10.後臺在Google上找到解決方案:
將https替換成http,
sudo gem sources -r http://rubygems.org
11.檢視映象
12.進入bin再執行:
./logstash-plugin install logstash-input-mongodb
13.安裝成功:
14.檢視外掛列表:
./logstash-plugin list
15.改回映象
sudo gem sources -a https://rubygems.org
16.新增logstash配置檔案
進入logstash-5.5.2/config
vim logstash.conf
input {
mongodb {
uri => 'mongodb://192.168.1.43:27017/testData'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name =>'testData.db'
collection => 'test_Current'
}
}
filter
{
# 把mongodb的_id替換掉,因為_id是跟es中的_id相沖突
mutate {
rename => ["_id", "uid"]
}
# ruby {
# code => "event.set('message', eval(event('title')))"
# }
}
output {
file {
path => "/var/log/mongons.log"
}
stdout {
codec => json_lines
}
elasticsearch {
hosts => ["192.168.1.171:9200"]
index => "testData"
manage_template=>true
document_type => "judicial"
}
}
17.啟動logstash
bin/logstash -f logstash.conf