EFK收集Kubernetes應用日誌
阿新 • • 發佈：2017-12-04
mark grace 修改 info mode 選項 進度 pri ng- 
本節內容：
- EFK介紹
- 安裝配置EFK
- 配置efk-rbac.yaml文件
- 配置 es-controller.yaml
- 配置 es-service.yaml
- 配置 fluentd-es-ds.yaml
- 配置 kibana-controller.yaml
- 配置 kibana-service.yaml
- 給 Node 設置標簽
- 執行定義文件
- 檢查執行結果
- 訪問 kibana
一、EFK介紹
- Logstash(或者Fluentd)負責收集日誌
- Elasticsearch存儲日誌並提供搜索
- Kibana負責日誌查詢和展示
官方地址：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
通過在每臺node上部署一個以DaemonSet方式運行的fluentd來收集每臺node上的日誌。Fluentd將docker日誌目錄/var/lib/docker/containers和/var/log目錄掛載到Pod中，然後Pod會在node節點的/var/log/pods目錄中創建新的目錄，可以區別不同的容器日誌輸出，該目錄下有一個日誌文件鏈接到/var/lib/docker/contianers目錄下的容器日誌輸出。
二、安裝配置EFK
1. 配置efk-rbac.yaml文件
EFK服務也需要一個efk-rbac.yaml文件，配置serviceaccount為efk。
[root@node1 opt]# mkdirefk [root@node1 opt]# cd efk
[root@node1 efk]# cat efk-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: efk namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: efk subjects: - kind: ServiceAccount name: efk namespace: kubeefk-rbac.yaml-system roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io
2. 配置 es-controller.yaml
[root@node1 efk]# vim es-controller.yaml apiVersion: v1 kind: ReplicationController metadata: name: elasticsearch-logging-v1 namespace: kube-system labels: k8s-app: elasticsearch-logging version: v1 kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: replicas: 2 selector: k8s-app: elasticsearch-logging version: v1 template: metadata: labels: k8s-app: elasticsearch-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccountName: efk containers: - image: index.tenxcloud.com/jimmy/elasticsearch:v2.4.1-2 name: elasticsearch-logging resources: # need more cpu upon initialization, therefore burstable class limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 9200 name: db protocol: TCP - containerPort: 9300 name: transport protocol: TCP volumeMounts: - name: es-persistent-storage mountPath: /data env: - name: "NAMESPACE" valueFrom: fieldRef: fieldPath: metadata.namespace volumes: - name: es-persistent-storage emptyDir: {}es-controller.yaml
3. 配置 es-service.yaml
[root@node1 efk]# vim es-service.yaml apiVersion: v1 kind: Service metadata: name: elasticsearch-logging namespace: kube-system labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "Elasticsearch" spec: ports: - port: 9200 protocol: TCP targetPort: db selector: k8s-app: elasticsearch-logginges-service.yaml
4. 配置 fluentd-es-ds.yaml
[root@node1 efk]# cat fluentd-es-ds.yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd-es-v1.22 namespace: kube-system labels: k8s-app: fluentd-es kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile version: v1.22 spec: template: metadata: labels: k8s-app: fluentd-es kubernetes.io/cluster-service: "true" version: v1.22 # This annotation ensures that fluentd does not get evicted if the node # supports critical pod annotation based priority scheme. # Note that this does not guarantee admission on the nodes (#40573). annotations: scheduler.alpha.kubernetes.io/critical-pod: ‘‘ spec: serviceAccountName: efk containers: - name: fluentd-es image: index.tenxcloud.com/jimmy/fluentd-elasticsearch:1.22 command: - ‘/bin/sh‘ - ‘-c‘ - ‘/usr/sbin/td-agent 2>&1 >> /var/log/fluentd.log‘ resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true nodeSelector: beta.kubernetes.io/fluentd-ds-ready: "true" tolerations: - key : "node.alpha.kubernetes.io/ismaster" effect: "NoSchedule" terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containersfluentd-es-ds.yaml
5. 配置 kibana-controller.yaml
[root@node1 efk]# cat kibana-controller.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kibana-logging namespace: kube-system labels: k8s-app: kibana-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: replicas: 1 selector: matchLabels: k8s-app: kibana-logging template: metadata: labels: k8s-app: kibana-logging spec: serviceAccountName: efk containers: - name: kibana-logging image: index.tenxcloud.com/jimmy/kibana:v4.6.1-1 resources: # keep request = limit to keep this container in guaranteed class limits: cpu: 100m requests: cpu: 100m env: - name: "ELASTICSEARCH_URL" value: "http://elasticsearch-logging:9200" - name: "KIBANA_BASE_URL" value: "/api/v1/proxy/namespaces/kube-system/services/kibana-logging" ports: - containerPort: 5601 name: ui protocol: TCPkibana-controller.yaml
6. 配置 kibana-service.yaml
[root@node1 efk]# cat kibana-service.yaml apiVersion: v1 kind: Service metadata: name: kibana-logging namespace: kube-system labels: k8s-app: kibana-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "Kibana" spec: ports: - port: 5601 protocol: TCP targetPort: ui selector: k8s-app: kibana-loggingkibana-service.yaml
root@node1 efk]# ls efk-rbac.yaml es-controller.yaml es-service.yaml fluentd-es-ds.yaml kibana-controller.yaml kibana-service.yaml
7. 給 Node 設置標簽
定義 DaemonSet fluentd-es-v1.22 時設置了 nodeSelector beta.kubernetes.io/fluentd-ds-ready=true ，所以需要在期望運行 fluentd 的 Node 上設置該標簽；
[root@node1 efk]# kubectl label nodes 172.16.7.151 beta.kubernetes.io/fluentd-ds-ready=true node "172.16.7.151" labeled [root@node1 efk]# kubectl label nodes 172.16.7.152 beta.kubernetes.io/fluentd-ds-ready=true node "172.16.7.152" labeled [root@node1 efk]# kubectl label nodes 172.16.7.153 beta.kubernetes.io/fluentd-ds-ready=true node "172.16.7.153" labeled
8. 執行定義文件
[root@node1 efk]# kubectl create -f .
9. 檢查執行結果
[root@node1 efk]# kubectl get deployment -n kube-system|grep kibana kibana-logging 1 1 1 1 1h [root@node1 efk]# kubectl get pods -n kube-system|grep -E ‘elasticsearch|fluentd|kibana‘ elasticsearch-logging-v1-nw3p3 1/1 Running 0 43m elasticsearch-logging-v1-pp89h 1/1 Running 0 43m fluentd-es-v1.22-cqd1s 1/1 Running 0 15m fluentd-es-v1.22-f5ljr 0/1 Error 6 15m fluentd-es-v1.22-x24jx 1/1 Running 0 15m kibana-logging-4293390753-kg8kx 1/1 Running 0 1h [root@node1 efk]# kubectl get service -n kube-system|grep -E ‘elasticsearch|kibana‘ elasticsearch-logging 10.254.50.63 <none> 9200/TCP 1h kibana-logging 10.254.169.159 <none> 5601/TCP 1h
kibana Pod 第一次啟動時會用較長時間(10-20分鐘)來優化和 Cache 狀態頁面，可以 tailf 該 Pod 的日誌觀察進度。
[root@node1 efk]# kubectl logs kibana-logging-4293390753-86h5d -n kube-system -f ELASTICSEARCH_URL=http://elasticsearch-logging:9200 server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-logging {"type":"log","@timestamp":"2017-10-13T00:51:31Z","tags":["info","optimize"],"pid":5,"message":"Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"} {"type":"log","@timestamp":"2017-10-13T01:13:36Z","tags":["info","optimize"],"pid":5,"message":"Optimization of bundles for kibana and statusPage complete in 1324.64 seconds"} {"type":"log","@timestamp":"2017-10-13T01:13:37Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:38Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:39Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:39Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:39Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:39Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:40Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:40Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-10-13T01:13:40Z","tags":["listening","info"],"pid":5,"message":"Server running at http://0.0.0.0:5601"} {"type":"log","@timestamp":"2017-10-13T01:13:45Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2017-10-13T01:13:49Z","tags":["status","plugin:[email protected]","info"],"pid":5,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
三、訪問kibana
1. 通過 kube-apiserver 訪問：獲取 kibana 服務 URL
[root@node1 efk]# kubectl cluster-info Kubernetes master is running at https://172.16.7.151:6443 Elasticsearch is running at https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging Heapster is running at https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/heapster Kibana is running at https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging KubeDNS is running at https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard monitoring-grafana is running at https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana monitoring-influxdb is running at https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump‘.
瀏覽器訪問 URL： https://172.16.7.151:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana
2. 通過 kubectl proxy 訪問：創建代理
[root@node1 efk]# kubectl proxy --address=‘172.16.7.151‘ --port=8086 --accept-hosts=‘^*$‘ &
瀏覽器訪問 URL：http://172.16.7.151:8086/api/v1/proxy/namespaces/kube-system/services/kibana-logging
如果你在這裏發現Create按鈕是灰色的無法點擊，且Time-filed name中沒有選項，fluentd要讀取/var/log/containers/目錄下的log日誌，這些日誌是從/var/lib/docker/containers/${CONTAINER_ID}/${CONTAINER_ID}-json.log鏈接過來的，查看你的docker配置，—-log-driver需要設置為json-file格式，默認的可能是journald。
查看當前的--log-driver:
[root@node1 ~]# docker version Client: Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-32.git88a4867.el7.centos.x86_64 Go version: go1.7.4 Git commit: 88a4867/1.12.6 Built: Mon Jul 3 16:02:02 2017 OS/Arch: linux/amd64 Server: Version: 1.12.6 API version: 1.24 Package version: docker-1.12.6-32.git88a4867.el7.centos.x86_64 Go version: go1.7.4 Git commit: 88a4867/1.12.6 Built: Mon Jul 3 16:02:02 2017 OS/Arch: linux/amd64 [root@node1 efk]# docker info |grep ‘Logging Driver‘ WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device. WARNING: bridge-nf-call-ip6tables is disabled Logging Driver: journald
修改當前版本docker的--log-driver:
[root@node1 ~]# vim /etc/sysconfig/docker OPTIONS=‘--selinux-enabled --log-driver=json-file --signature-verification=false‘ [root@node1 efk]# systemctl restart docker
【註意】：本來修改這個參數應該在在/etc/docker/daemon.json文件中添加:
{ "log-driver": "json-file", }
但是在該版本中，--log-driver是在文件/etc/sysconfig/docker中定義的。在docker-ce版本中，默認的--log-driver是json-file。
遇到的問題：
由於之前在/etc/docker/daemon.json中配置--log-driver，重啟導致docker程序啟動失敗，等到後來在/etc/sysconfig/docker配置文件中配置好後，啟動docker卻發現當前node變成NotReady狀態，所有的Pod也變為Unknown狀態。查看kubelet狀態，發現kubelet程序已經掛掉了。
[root@node1 ~]# kubectl get nodes NAME STATUS AGE VERSION 172.16.7.151 NotReady 28d v1.6.0 172.16.7.152 Ready 28d v1.6.0 172.16.7.153 Ready 28d v1.6.0
啟動kubelet：
[root@node1 ~]# systemctl start kubelet [root@node1 ~]# kubectl get nodes NAME STATUS AGE VERSION 172.16.7.151 Ready 28d v1.6.0 172.16.7.152 Ready 28d v1.6.0 172.16.7.153 Ready 28d v1.6.0
瀏覽器再次訪問 kibana URL：http://172.16.7.151:8086/api/v1/proxy/namespaces/kube-system/services/kibana-logging，此時就會發現有Create按鈕了。
在 Settings -> Indices 頁面創建一個 index（相當於 mysql 中的一個 database），去掉已經勾選的 Index contains time-based events，使用默認的 logstash-* pattern，點擊 Create ;
創建Index後，可以在 Discover 下看到 ElasticSearch logging 中匯聚的日誌。
EFK收集Kubernetes應用日誌