组件自带metrics接口,且有ServiceMonitor,但没有service和endpoint:
二进制安装的kube-controller-manager和kube-scheduler无法获取metrics(kubeadm方式不会出现此问题)
查看问题:
寻找kube-controller-manager和kube-scheduler的红色的告警
controller-manager监控
1、因为controller监听了127.0.0.1,所以监听地址需要修改成全网段
2、prometheus自动注册了targets相关监控项,但是没有对应标签的service
kubectl -n monitoring get servicemonitor
kubectl -n monitoring get servicemonitor kube-controller-manager -o yaml | less
可查出,这个监控项在kube-system的ns中找一个标签为k8s-app: kube-controller-manager的service但是kube-system没有这个service
处理问题
所有master修改:
vim /usr/lib/systemd/system/kube-controller-manager.service
创建endpoint和svc,名称保持一致(kube-controller-manage-monitor)
vim endpoint-svc.yaml
apiVersion: v1
items:
- apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: kube-controller-manager # svc添加label,需要和servicemonitor寻找的一致
name: kube-controller-manage-monitor
namespace: kube-system # 需要和servicemonitor寻找的名称空间一致
spec:
ports:
- name: http-metrics # 端口名称需要匹配servicemonitor寻找的名称
port: 10252 # 监听kube-controller-manager暴露的10252端口
protocol: TCP
targetPort: 10252
type: ClusterIP
- apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: kube-controller-manager # endpoint添加label
name: kube-controller-manage-monitor #名称与service一致
namespace: kube-system
subsets:
- addresses:
- ip: 192.168.1.10 # 监听节点
- ip: 192.168.1.11
- ip: 192.168.1.12
ports:
- name: http-metrics # 端口名称需要匹配servicemonitor寻找的名称
port: 10252 # 端口一致
protocol: TCP
kind: List
metadata:
resourceVersion: ""
selfLink: ""
kubectl create -f endpoint-svc.yaml
成功获取metrics
kube-scheduler监控
同理,所有master修改:
vim /usr/lib/systemd/system/kube-scheduler.service
systemctl daemon-reload
systemctl restart kube-scheduler.service
vim endpoint-svc-scheduler.yaml
apiVersion: v1
items:
- apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: kube-scheduler # svc添加label
name: kube-scheduler-monitor
namespace: kube-system
spec:
ports:
- name: http-metrics # 端口名称需要匹配
port: 10251 # 监听kube-scheduler暴露的10252端口
protocol: TCP
targetPort: 10251
type: ClusterIP
- apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: kube-scheduler # endpoint添加label
name: kube-scheduler-monitor #名称与service一致
namespace: kube-system
subsets:
- addresses:
- ip: 192.168.1.10 # 监听节点
- ip: 192.168.1.11
- ip: 192.168.1.12
ports:
- name: http-metrics # 端口名称需要匹配
port: 10251 # 端口一致
protocol: TCP
kind: List
metadata:
resourceVersion: ""
selfLink: ""
kubectl create -f endpoint-svc-scheduler.yaml
组件自带metrics接口,但没有ServiceMonitor
etcd监控
获取etcd的metrics:
curl https://192.168.1.10:2379/metrics --cert /etc/etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem -k
尝试创建endpoint+service连接到这个端口,然后使用service获取到metrics
vim etcd-svc-edpoint.yml
apiVersion: v1
items:
- apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: etcd
name: etcd-monitor
namespace: kube-system
spec:
ports:
- name: etcd
port: 2379
protocol: TCP
targetPort: 2379
type: ClusterIP
- apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: etcd
name: etcd-monitor
namespace: kube-system
subsets:
- addresses:
- ip: 192.168.1.10
- ip: 192.168.1.11
- ip: 192.168.1.12
ports:
- name: etcd
port: 2379
protocol: TCP
kind: List
metadata:
resourceVersion: ""
selfLink: ""
kubectl create -f etcd-svc-edpoint.yml
使用创建的svc地址测试下metrics
curl https://10.102.133.91:2379/metrics --cert /etc//etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem -k
生成secret形式证书:
kubectl -n monitoring create secret generic etcd-certs --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem --from-file=/etc/etcd/ssl/etcd-ca.pem
创建etcd的ServiceMonitor
vim etcd-serviceMonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: etcd
name: etcd
namespace: monitoring
spec:
endpoints:
- interval: 30s
port: etcd # 名字对应
scheme: https # 需要配置成https证书形式
tlsConfig:
caFile: /etc/prometheus/secrets/etcd-certs/etcd-ca.pem
certFile: /etc/prometheus/secrets/etcd-certs/etcd.pem
keyFile: /etc/prometheus/secrets/etcd-certs/etcd-key.pem
insecureSkipVerify: true
namespaceSelector:
matchNames:
- kube-system # 名称空间对应
selector:
matchLabels:
k8s-app: etcd # 与etcd service相匹配
修改kube-prometheus/manifests下的prometheus-prometheus.yaml
vim prometheus-prometheus.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
image: quay.io/prometheus/prometheus:v2.15.2
nodeSelector:
kubernetes.io/os: linux
podMonitorNamespaceSelector: {}
podMonitorSelector: {}
replicas: 1
resources:
requests:
memory: 400Mi
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.15.2
secrets: # 添加证书配置
- etcd-certs
kubectl replace -f prometheus-prometheus.yaml
成功获取
可以添加一个dashboard https://grafana.com/grafana/dashboards/3070-etcd/
导入json
组件没有metrics接口
如果是非云原生应用组件,开发时可能没有metrics接口,只能使用第三方exporter(或exporter插件)定期去采集相关数据
可以通过github直接寻找中间件的exporter,下面以这个kafka为例
https://github.com/search?q=exporter&type=repositories
先部署一套kafka集群,依据https://www.ljh.cool/37543.html文档方式部署
kubectl get po -n public-service
直接将镜像部署到k8s中
vim kafka-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app: kafka-exporter
name: kafka-exporter
namespace: monitoring
spec:
selector:
matchLabels:
app: kafka-exporter
template:
metadata:
labels:
app: kafka-exporter
spec:
containers:
- args:
- --kafka.server=kafka-0.kafka-headless.public-service:9092
env:
- name: TZ
value: Asia/Shanghai
- name: LANG
value: C.UTF-8
image: danielqsj/kafka-exporter:latest
imagePullPolicy: IfNotPresent
name: kafka-exporter
ports:
- containerPort: 9308
name: web
protocol: TCP
volumeMounts:
- mountPath: /usr/share/zoneinfo/Asia/Shanghai
name: tz-config
- mountPath: /etc/localtime
name: tz-config
- mountPath: /etc/timezone
name: timezone
dnsPolicy: ClusterFirst
volumes:
- hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
type: ""
name: tz-config
- hostPath:
path: /etc/timezone
type: ""
name: timezone
kubectl create -f kafka-exporter.yaml
kubectl get deployments.apps -n monitoring
查看pod日志
kubectl -n monitoring logs -f kafka-exporter-74cc5d69c6-vmwvc
vim kafka-exporter-svc.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-exporter
name: kafka-exporter-monitor
namespace: monitoring
spec:
ports:
- name: container-port
port: 9308
protocol: TCP
targetPort: 9308
selector:
app: kafka-exporter
type: ClusterIP
kubectl create -f kafka-exporter-svc.yaml
kubectl get svc -n monitoring
curl 10.109.248.95:9308/metrics
serviceMonitor注册
vim kafka-serviceMonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: kafka-exporter
name: kafka-exporter
namespace: monitoring
spec:
endpoints:
- interval: 30s
port: container-port
namespaceSelector:
matchNames:
- monitoring
selector:
matchLabels:
app: kafka-exporter
添加已经提供的dashboard
发布者:LJH,转发请注明出处:https://www.ljh.cool/37883.html
评论列表(2条)
Everything is very open with a precise description of the issues. It was really informative. Your website is very helpful. Many thanks for sharing!
@נערות ליווי באילת:Thank you for your recognition!