技術標籤：pip install requestsrequests content decoderequests payload

閱讀提示｜本文大概1700字 閱讀需要20分鐘

背景

• 對於某些應用，需要用於動態擴充套件的指標不僅僅是 CPU/Mem，有時候還需要用到自定義度量，比如 http_requests。

• 在 OpenShift 4 目前的版本(OCP 4.3)裡，exposing custom application metrics for HPA 功能還屬於 Technology Preview feature only。

• 但這不妨礙我們功能上的實現，我們可以直接利用 OpenShift 4 OperatorHub 中的 Prometheus Operator 來實現。

實現過程

1. 建立執行 Prometheus Operator 的 namespace

oc new-project ns1

2. 使用 OpenShift 4 的 UI 來部署 Prometheus Operator

• 使用 UI 部署 Prometheus Operator

  • 確保是安裝到我們的目標 namespace -> ns1

  • UI -> Administrator page -> OperatorHub -> Prometheus Operator -> install

• 部署 Prometheus instance

  • 直接使用 UI 中預設的 yaml 建立即可

  • UI -> Administrator page -> Installed Operators -> Prometheus Operator -> Create Instance

3. 建立 Prometheus instance 的 route

• 主要是可以用於後面部署好測試應用之後可以在 Prometheus 的 UI 上驗證我們的部署是否成功

  • 可以在頁面上關注 Targets，以及在 Graph 執行查詢

oc expose svc prometheus-operated -n ns1

4. 建立該 Prometheus 需要的 RBAC 以及資源物件

• 查詢 OpenShift 4 中自帶的 prometheus-adapter 使用的 image，用於建立我們自定義的 adapter

oc get -n openshift-monitoring deploy/prometheus-adapter -o jsonpath="{..image}"
 

使用下面的 yaml 來建立對應的 RBAC 以及 Objects，注意替換上一步得到的 image 路徑。

• 下面的 yaml 是可以根據不同的環境做修改的，主要修改點是 namespace 以及 ConfigMap

  • ConfigMap 的內容主要是定義配置如何從 prometheus 獲取資料，並與 Kubernetes 的資源做對應，以及如何在 api 介面中展示

cat << EOF > deploy.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
  name: custom-metrics-apiserver
  namespace: ns1---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-server-resources
rules:- apiGroups:- custom.metrics.k8s.io
  resources: ["*"]
  verbs: ["*"]---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-resource-reader
rules:- apiGroups:- ""
  resources:- namespaces- pods- services
  verbs:- get- list---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: ns1---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: custom-metrics-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: ns1---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-resource-reader
subjects:- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: ns1---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hpa-controller-custom-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-server-resources
subjects:- kind: ServiceAccount
  name: horizontal-pod-autoscaler
  namespace: kube-system---
apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: ns1
data:
  config.yaml: |
    rules:- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)_seconds_total$as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters:- isNot: ^container_.*_seconds_total$
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)_total$as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters:- isNot: ^container_.*_total$
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)$as: ""
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:- isNot: .*_total$
      resources:
        template: <<.Resource>>
      name:
        matches: ""as: ""
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:- isNot: .*_seconds_total
      resources:
        template: <<.Resource>>
      name:
        matches: ^(.*)_total$as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters: []
      resources:
        template: <<.Resource>>
      name:
        matches: ^(.*)_seconds_total$as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
    resourceRules:
      cpu:
        containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
        nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
        resources:
          overrides:
            instance:
              resource: node
            namespace:
              resource: namespace
            pod_name:
              resource: pod
        containerLabel: container_name
      memory:
        containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>)
        nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
        resources:
          overrides:
            instance:
              resource: node
            namespace:
              resource: namespace
            pod_name:
              resource: pod
        containerLabel: container_name
      window: 1m---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.openshift.io/serving-cert-secret-name: prometheus-adapter-tls
  labels:
    name: prometheus-adapter
  name: prometheus-adapter
  namespace: ns1
spec:
  ports:- name: https
    port: 443
    targetPort: 6443
  selector:
    app: prometheus-adapter
  type: ClusterIP---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: prometheus-adapter
    namespace: ns1
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-adapter
  name: prometheus-adapter
  namespace: ns1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-adapter
  template:
    metadata:
      labels:
        app: prometheus-adapter
      name: prometheus-adapter
    spec:
      serviceAccountName: custom-metrics-apiserver
      containers:- name: prometheus-adapter
        image: <上一步得到的image替換這裡>
        args:- --secure-port=6443- --tls-cert-file=/var/run/serving-cert/tls.crt- --tls-private-key-file=/var/run/serving-cert/tls.key- --logtostderr=true- --prometheus-url=http://prometheus-operated.ns1.svc:9090/- --metrics-relist-interval=1m- --v=4- --config=/etc/adapter/config.yaml
        ports:- containerPort: 6443
        volumeMounts:- mountPath: /var/run/serving-cert
          name: volume-serving-cert
          readOnly: true- mountPath: /etc/adapter/
          name: config
          readOnly: true- mountPath: /tmp
          name: tmp-vol
      volumes:- name: volume-serving-cert
        secret:
          secretName: prometheus-adapter-tls- name: config
        configMap:
          name: adapter-config- name: tmp-vol
        emptyDir: {}---EOF
# 建立
oc apply -f deploy.yaml

驗證上一步我們建立的物件，比如 api

oc get apiservice v1beta1.custom.metrics.k8s.io

到這裡，我們的部署工作基本完成了，剩下的就是使用應用來驗證基於 http_requests 的 HPA 了

5. 建立測試應用

• 我們使用一個新的 namespace 來部署應用

  • 直接使用下邊的 yaml 檔案來建立即可，包括 my-new-hpa 這個 namespace

cat << EOF > prometheus-example-app.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: my-new-hpa---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-example-app
  name: prometheus-example-appnamespace: my-new-hpa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-example-apptemplate:
    metadata:
      labels:
        app: prometheus-example-app
    spec:
      containers:- image: quay.io/brancz/prometheus-example-app:v0.2.0
        imagePullPolicy: IfNotPresent
        name: prometheus-example-app---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus-example-app
  name: prometheus-example-appnamespace: my-new-hpa
spec:
  ports:- port: 8080
    protocol: TCP
    targetPort: 8080
    name: web
  selector:
    app: prometheus-example-app
  type: ClusterIPEOF
# 建立應用
oc apply -f prometheus-example-app.yaml

建立 ServiceMonitor

# 建立yaml檔案
cat << EOF > example-app-service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: prometheus-example-monitor
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - interval: 30s
    port: web
    scheme: http
  namespaceSelector:
    matchNames:
    - my-new-hpa    
  selector:
    matchLabels:
      app: prometheus-example-app
EOF
# 建立ServiceMonitor
oc apply -f example-app-service-monitor.yaml

給 Prometheus 訪問新 namespace 的許可權

echo "---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-new-hpa-rolebinding
  namespace: my-new-hpa
subjects:
  - kind: ServiceAccount
    name: prometheus-k8s
    namespace: ns1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view" | oc create -f -

在 Prometheus UI 上查詢 http_request 指標，或者通過 api 驗證

# Prometheus UI
http_requests_total{job="prometheus-example-app"}# OpenShift api
oc get --raw /apis/custom.metrics.k8s.io/v1beta1/ | jq -r '.resources[] | select(.name | contains("pods/http"))'

建立我們應用的 HPA

echo "---
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: pod-autoscale-custom
  namespace: my-new-hpa
spec:
  scaleTargetRef:
    kind: Deployment
    name: prometheus-example-app
    ## apiVersion: apps.openshift.io/v1 這個api沒有deployment，所以需要使用extensions/v1beta1這個api
    apiVersion: extensions/v1beta1
  minReplicas: 1
  maxReplicas: 4
  metrics:
    - type: Pods
      pods:
        metricName: http_requests
        targetAverageValue: 300m" | oc create -f -

給應用施加壓力並觀察應用例項數量是否隨著 http_requests 壓力的增加而擴充套件

oc expose service prometheus-example-app -n my-new-hpaAUTOSCALE_ROUTE=$(oc get route prometheus-example-app -n my-new-hpa -o jsonpath='{ .spec.host}')while true;do curl http://$AUTOSCALE_ROUTE;sleep .5;done
oc describe hpa pod-autoscale-custom -n my-new-hpa
oc get pods -n my-new-hpa

我們預期是 pod 的數量會隨著 http_requests 的增加而擴充套件到 4 個，停止加壓後一段時間又恢復到 1 個 pod