k8s监控篇-02 prometheus处理监控组件自带metrics和非自带metrics监控问题

组件自带metrics接口,且有ServiceMonitor,但没有service和endpoint:

二进制安装的kube-controller-manager和kube-scheduler无法获取metrics(kubeadm方式不会出现此问题)

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

查看问题:

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

寻找kube-controller-manager和kube-scheduler的红色的告警

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

controller-manager监控

1、因为controller监听了127.0.0.1,所以监听地址需要修改成全网段

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

2、prometheus自动注册了targets相关监控项,但是没有对应标签的service

kubectl -n monitoring get servicemonitor

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

kubectl -n monitoring get servicemonitor kube-controller-manager -o yaml | less

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

可查出,这个监控项在kube-system的ns中找一个标签为k8s-app: kube-controller-manager的service但是kube-system没有这个service

处理问题

所有master修改:

vim /usr/lib/systemd/system/kube-controller-manager.service

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

创建endpoint和svc,名称保持一致(kube-controller-manage-monitor)

vim endpoint-svc.yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      k8s-app: kube-controller-manager # svc添加label,需要和servicemonitor寻找的一致
    name: kube-controller-manage-monitor
    namespace: kube-system # 需要和servicemonitor寻找的名称空间一致
  spec:
    ports:
    - name: http-metrics # 端口名称需要匹配servicemonitor寻找的名称
      port: 10252 # 监听kube-controller-manager暴露的10252端口
      protocol: TCP
      targetPort: 10252
    type: ClusterIP
- apiVersion: v1
  kind: Endpoints
  metadata:
    labels:
      k8s-app: kube-controller-manager # endpoint添加label
    name: kube-controller-manage-monitor #名称与service一致
    namespace: kube-system
  subsets:
  - addresses:
    - ip: 192.168.1.10 # 监听节点
    - ip: 192.168.1.11
    - ip: 192.168.1.12
    ports:
    - name: http-metrics  # 端口名称需要匹配servicemonitor寻找的名称
      port: 10252 # 端口一致
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

kubectl create -f endpoint-svc.yaml

成功获取metrics

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

kube-scheduler监控

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

同理,所有master修改:

vim /usr/lib/systemd/system/kube-scheduler.service

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

systemctl daemon-reload
systemctl restart kube-scheduler.service 

vim endpoint-svc-scheduler.yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      k8s-app: kube-scheduler # svc添加label
    name: kube-scheduler-monitor
    namespace: kube-system
  spec:
    ports:
    - name: http-metrics # 端口名称需要匹配
      port: 10251 # 监听kube-scheduler暴露的10252端口
      protocol: TCP
      targetPort: 10251
    type: ClusterIP
- apiVersion: v1
  kind: Endpoints
  metadata:
    labels:
      k8s-app: kube-scheduler # endpoint添加label
    name: kube-scheduler-monitor #名称与service一致
    namespace: kube-system
  subsets:
  - addresses:
    - ip: 192.168.1.10 # 监听节点
    - ip: 192.168.1.11
    - ip: 192.168.1.12
    ports:
    - name: http-metrics  # 端口名称需要匹配
      port: 10251 # 端口一致
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

kubectl create -f endpoint-svc-scheduler.yaml

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

组件自带metrics接口,但没有ServiceMonitor

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

etcd监控

获取etcd的metrics:

curl https://192.168.1.10:2379/metrics --cert /etc/etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem -k

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

尝试创建endpoint+service连接到这个端口,然后使用service获取到metrics

vim etcd-svc-edpoint.yml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      k8s-app: etcd
    name: etcd-monitor
    namespace: kube-system
  spec:
    ports:
    - name: etcd
      port: 2379
      protocol: TCP
      targetPort: 2379
    type: ClusterIP

- apiVersion: v1
  kind: Endpoints
  metadata:
    labels:
      k8s-app: etcd
    name: etcd-monitor
    namespace: kube-system
  subsets:
  - addresses:
    - ip: 192.168.1.10
    - ip: 192.168.1.11
    - ip: 192.168.1.12
    ports:
    - name: etcd
      port: 2379
      protocol: TCP
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

kubectl create -f etcd-svc-edpoint.yml

使用创建的svc地址测试下metrics

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

curl https://10.102.133.91:2379/metrics --cert /etc//etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem -k

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

生成secret形式证书:

kubectl -n monitoring create secret generic etcd-certs --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem --from-file=/etc/etcd/ssl/etcd-ca.pem

创建etcd的ServiceMonitor

vim etcd-serviceMonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd
  name: etcd
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: etcd # 名字对应
    scheme: https # 需要配置成https证书形式
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-certs/etcd-ca.pem
      certFile: /etc/prometheus/secrets/etcd-certs/etcd.pem
      keyFile: /etc/prometheus/secrets/etcd-certs/etcd-key.pem
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - kube-system # 名称空间对应
  selector:
    matchLabels:
      k8s-app: etcd # 与etcd service相匹配

修改kube-prometheus/manifests下的prometheus-prometheus.yaml

vim prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.15.2
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 1
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.15.2
  secrets: # 添加证书配置
  - etcd-certs

kubectl replace -f prometheus-prometheus.yaml

成功获取

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

可以添加一个dashboard https://grafana.com/grafana/dashboards/3070-etcd/

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

导入json

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

组件没有metrics接口

如果是非云原生应用组件,开发时可能没有metrics接口,只能使用第三方exporter(或exporter插件)定期去采集相关数据

可以通过github直接寻找中间件的exporter,下面以这个kafka为例

https://github.com/search?q=exporter&type=repositories

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

先部署一套kafka集群,依据https://www.ljh.cool/37543.html文档方式部署

kubectl get po -n public-service

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

直接将镜像部署到k8s中

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

vim kafka-exporter.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: kafka-exporter
  name: kafka-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: kafka-exporter
  template:
    metadata:
      labels:
        app: kafka-exporter
    spec:
      containers:
      - args:
        - --kafka.server=kafka-0.kafka-headless.public-service:9092
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: danielqsj/kafka-exporter:latest
        imagePullPolicy: IfNotPresent
        name: kafka-exporter
        ports:
        - containerPort: 9308
          name: web
          protocol: TCP
        volumeMounts:
        - mountPath: /usr/share/zoneinfo/Asia/Shanghai
          name: tz-config
        - mountPath: /etc/localtime
          name: tz-config
        - mountPath: /etc/timezone
          name: timezone
      dnsPolicy: ClusterFirst
      volumes:
      - hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
          type: ""
        name: tz-config
      - hostPath:
          path: /etc/timezone
          type: ""
        name: timezone

kubectl create -f kafka-exporter.yaml

kubectl get deployments.apps -n monitoring

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

查看pod日志

kubectl -n monitoring logs -f kafka-exporter-74cc5d69c6-vmwvc

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

vim kafka-exporter-svc.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-exporter
  name: kafka-exporter-monitor
  namespace: monitoring
spec:
  ports:
  - name: container-port
    port: 9308
    protocol: TCP
    targetPort: 9308
  selector:
    app: kafka-exporter
  type: ClusterIP

kubectl create -f kafka-exporter-svc.yaml

kubectl get svc -n monitoring

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

curl 10.109.248.95:9308/metrics

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

serviceMonitor注册

vim kafka-serviceMonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kafka-exporter
  name: kafka-exporter
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: container-port
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      app: kafka-exporter
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

添加已经提供的dashboard

k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题
k8s监控篇-02  prometheus处理监控组件自带metrics和非自带metrics监控问题

发布者:LJH,转发请注明出处:https://www.ljh.cool/37883.html

(0)
上一篇 2023年8月22日 上午10:44
下一篇 2023年9月2日 下午2:28

相关推荐

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注

评论列表(2条)