Day21，Prometheus Operator

正文

今天要使用helm来安装 Prometheus Operator，原先的prometheus operator chart已经被弃用，目前转为新的prometheus-community/kube-prometheus-stack，而kube-prometheus就是基於prometheus + prometheus operator上的设定与布署并整合在kubernetes上。

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create ns monitoring

helm install homelab-monitoring prometheus-community/kube-prometheus-stack --version 18.0.8 -n monitoring

可以设定的实在太多，就从预设的安装来反看吧。

安装的instance，简单来说就是prometheus、prometheus operator、alertmanager、grafana

另外也会根据prometheus operator产出crd，以及不少的custom resources

举例prometheuses的custom resource来看

kubectl get prometheuses.monitoring.coreos.com homelab-monitoring-kube-pr-prometheus -o yaml  -n monitoring

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    meta.helm.sh/release-name: homelab-monitoring
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2021-09-15T13:39:16Z"
  generation: 1
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: homelab-monitoring
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 18.0.8
    chart: kube-prometheus-stack-18.0.8
    heritage: Helm
    release: homelab-monitoring
  name: homelab-monitoring-kube-pr-prometheus
  namespace: monitoring
  resourceVersion: "304609"
  uid: 8ebae130-9d60-462c-bdd7-be25853c1754
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: homelab-monitoring-kube-pr-alertmanager
      namespace: monitoring
      pathPrefix: /
      port: web
  enableAdminAPI: false
  externalUrl: http://homelab-monitoring-kube-pr-prometheus.monitoring:9090
  image: quay.io/prometheus/prometheus:v2.28.1
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      release: homelab-monitoring
  portName: web
  probeNamespaceSelector: {}
  probeSelector:
    matchLabels:
      release: homelab-monitoring
  replicas: 1
  retention: 10d
  routePrefix: /
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      app: kube-prometheus-stack
      release: homelab-monitoring
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: homelab-monitoring-kube-pr-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: homelab-monitoring
  shards: 1
  version: v2.28.1

从这份文档当中，我们可以大概得知的设定如资料保存为预设10天 ，且由於我们一开始并没有设定任何参数
像podmonitor、servicemonitor、rules、probe 这些在custom resources建立时都需要matchLabels release: homelab-monitoring，所在namespaces的选择则是不受限制。

继续看一下预设帮忙产出的metrics service

可以看到servicemonitor也有被建立

将prometheus port-forward到本机看看

kubectl port-forward svc/homelab-monitoring-kube-pr-prometheus 9090:9090 -n monitoring

可以看到kube-proxy、kube-controller-manager状态为dwon，kube-proxy是因为我们只有开启127.0.0.1的服务监听，导致使用IP时无法存取，修改过後便能够使用。而kube-controller-manager则是因为设定上关闭了http的存取，仅能使用身份验证的https，这边都先想办法让他打开，再检查後如下

这边认真查了一下1.20的文件kube-controller-manager上面已经找不到--port=0，而port号使用也从10252修改为10257，但透过kubespray布署的static pod上面仍存在--port=0这段设定，索性把他注解掉之後发现还是有作用的，10252 Port又出来了listen了。

我的感觉是，看起来在後续的版本这个helm chart这边都会需要再做调整(完全弃用时)，而servicemonitor存取的metrics service这块最终看起来都会希望是以https+serviceaccount(authorized)作为较高安全性的考量来存取。

可以看到预设已带入非常多dashboard

到这边就可以根据需求在grafana上做一些可观测性的工作

接着看看 alertmanagers.monitoring.coreos.com 这边的alertmanager custom resource，如下所示，alertmanagerConfigNamespaceSelector: {}& alertmanagerConfigSelector: {}
预设就是全部的namespaces且不限制config

apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Alertmanager
  metadata:
    annotations:
      meta.helm.sh/release-name: homelab-monitoring
      meta.helm.sh/release-namespace: monitoring
    creationTimestamp: "2021-09-15T13:39:16Z"
    generation: 1
    labels:
      app: kube-prometheus-stack-alertmanager
      app.kubernetes.io/instance: homelab-monitoring
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/part-of: kube-prometheus-stack
      app.kubernetes.io/version: 18.0.8
      chart: kube-prometheus-stack-18.0.8
      heritage: Helm
      release: homelab-monitoring
    name: homelab-monitoring-kube-pr-alertmanager
    namespace: monitoring
    resourceVersion: "304594"
    uid: 5f05b7a3-ed7f-43bf-b1b1-be6e907aa187
  spec:
    alertmanagerConfigNamespaceSelector: {}
    alertmanagerConfigSelector: {}
    externalUrl: http://homelab-monitoring-kube-pr-alertmanager.monitoring:9093
    image: quay.io/prometheus/alertmanager:v0.22.2
    listenLocal: false
    logFormat: logfmt
    logLevel: info
    paused: false
    portName: web
    replicas: 1
    retention: 120h
    routePrefix: /
    securityContext:
      fsGroup: 2000
      runAsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000
    serviceAccountName: homelab-monitoring-kube-pr-alertmanager
    version: v0.22.2
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

而设定alermangerconfig的方式也与一般alertmanager相同，可以参考连结，最後这边就不继续设定下去罗。

闲聊

今天做LAB的时候才赫然发现，以前在用的prometheus operator竟然弃用了而我却浑然不知，看来真的是太久没装了QQ，还有昨天loki-stack的Grafana 版本7.x，今天的kube-prometheus Grafana 版本8.x，登入页面长不一样带给我的冲击都好大唷XD。