Filebeat Nginx Module 自定義欄位
阿新 • • 發佈:2021-06-10
Filebeat Nginx Module 自定義欄位
一、修改/usr/local/nginx/conf/nginx.conf中
log_format access '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" ' '"$http_x_real_ip" "$server_addr" "$host" ' '$request_time $upstream_response_time "$upstream_addr" ' '"$time_iso8601"'; access_log /var/log/nginx/access.log access;
二、啟用filebeat的nginx module
filebeat modules enable nginx
三、filebeat nginx module 增加欄位
檔案 /usr/share/filebeat/module/nginx/access/ingest/default.json 中
原始:
"grok": { "field": "message", "patterns":[ "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\"" ], "pattern_definitions": { "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*" }
修改後:
"grok": { "field": "message", "patterns":[ "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.xff}\" \"%{DATA:nginx.access.x_real_ip}\" \"%{DATA:nginx.access.server_addr}\" \"%{DATA:nginx.access.host}\" %{DATA:nginx.access.request_time} %{DATA:nginx.access.upstream_response_time} \"%{DATA:nginx.access.upstream_addr}\" \"%{DATA:nginx.access.time_iso8601}\"" ], "pattern_definitions": { "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*" }
四、檔案 /etc/filebeat/fields.yml 中
找到nginx欄位配置
在
- name: agent
type: text
description: >
Contains the un-parsed user agent string. Only present if the user
agent Elasticsearch plugin is not available or not used.
後面加入
- name: xff
type: group
description: >
http_x_forwarded_for.
- name: x_real_ip
type: group
description: >
http_x_real_ip.
- name: server_addr
type: group
description: >
server_addr 伺服器地址.
- name: host
type: group
description: >
host http_host http主機頭.
- name: request_time
type: group
description: >
request_time 請求時間.
- name: upstream_response_time
type: group
description: >
upstream_response_time 後端響應時間.
- name: upstream_addr
type: group
description: >
upstream_addr 後端地址.
- name: time_iso8601
type: group
description: >
time_iso8601 iso8601格式時間.
五、Filebeta中 nginx 日誌路徑定義
在檔案 /etc/filebeat/modules.d/nginx.yml 中修改日誌路徑
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/data/wwwlogs/*.log*"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/usr/local/nginx/logs/error.log*"]