day20 : redisDB keyDB on K8S (下)

昨天简略介绍了redis cluster的架构以及小小的讲了一下keydb，所以今天会透过redis operator的方式布署redis cluster然後用helm的方式布署keyDB，同时会布署两套keyDB，一套做为独立使用，另一套去sync redis。

curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.18.3/install.sh | bash -s v0.18.3

kubectl create -f https://operatorhub.io/install/redis-operator.yaml

透过这两个指令完成redis 的operator，接着配置合适的pv 和redis-passwd 给要建立cluster 的namespaces，再配置redis cluster yaml，就可以建出redis cluster罗

apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: RedisCluster
metadata:
  name: redis-cluster
spec:
  clusterSize: 3
  kubernetesConfig:
    image: 'quay.io/opstree/redis:v6.2'
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
    redisSecret:
      name: redis-secret
      key: password
    serviceType: ClusterIP
  redisExporter:
    enabled: true
    image: 'quay.io/opstree/redis-exporter:1.0'
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
  redisLeader:
    serviceType: ClusterIP
  redisFollower:
    serviceType: ClusterIP
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

另外顺便提供单机版本

apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: Redis
metadata:
  name: redis-standalone
spec:
  redisExporter:
    enabled: true
    image: 'quay.io/opstree/redis-exporter:1.0'
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
  kubernetesConfig:
    image: 'quay.io/opstree/redis:v6.2'
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
    redisSecret:
      name: redis-secret
      key: password
    serviceType: LoadBalancer
  redisConfig: {}
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

接着来建置一套keyDB

helm repo add enapter https://enapter.github.io/charts/
helm install my-keydb enapter/keydb --version 0.23.0 -n demo

两套都是使用6379以及redis-cli就可以操作使用了。

接着为了实现keyDB与redis的结合，要使用keyDB中的两个参数
active-replica yes
replicaof redis-cluster-leader 6379
第一个是主动同步
第二个则是要同步的主机IP因为使用k8s所以用service
这个要设计在keyDB的config上，所以要对sts的档案做一些修改，

这个是会使用到的sts yaml，其中有使用一个secret档案my-keydb-utils

spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: my-keydb
      app.kubernetes.io/name: keydb
  serviceName: my-keydb-headless
  template:
    metadata:
      annotations:
        checksum/secret-utils: df252d80f012eac3f195684b8d022babb3a58b7701a528aa8971920f9cee350a
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: my-keydb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: keydb
        app.kubernetes.io/version: 6.0.18
        helm.sh/chart: keydb-0.23.0
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - keydb
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                  - my-keydb
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - command:
        - /utils/server.sh
        image: eqalpha/keydb:x86_64_v6.0.18
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: keydb
          timeoutSeconds: 1
        name: keydb
        ports:
        - containerPort: 6379
          name: keydb
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: keydb
          timeoutSeconds: 1
        resources: {}
        startupProbe:
          failureThreshold: 30
          periodSeconds: 5
          successThreshold: 1
          tcpSocket:
            port: keydb
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: keydb-data
        - mountPath: /utils
          name: utils
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: utils
        secret:
          defaultMode: 493
          items:
          - key: server.sh
            path: server.sh
          secretName: my-keydb-utils
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: keydb-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      volumeMode: Filesystem
    status:
      phase: Pending

apiVersion: v1
data:
  server.sh: IyEvYmluL2Jhc2gKc2V0IC1ldXhvIHBpcGVmYWlsCgpob3N0PSIkKGhvc3RuYW1lKSIKcG9ydD0iNjM3OSIKcmVwbGljYXM9KCkKZm9yIG5vZGUgaW4gezAuLjJ9OyBkbwogIGlmIFsgIiRob3N0IiAhPSAibXkta2V5ZGItJHtub2RlfSIgXTsgdGhlbgogICAgICByZXBsaWNhcys9KCItLXJlcGxpY2FvZiBteS1rZXlkYi0ke25vZGV9Lm15LWtleWRiLWhlYWRsZXNzICR7cG9ydH0iKQogIGZpCmRvbmUKZXhlYyBrZXlkYi1zZXJ2ZXIgL2V0Yy9rZXlkYi9yZWRpcy5jb25mIFwKICAgIC0tYWN0aXZlLXJlcGxpY2EgeWVzIFwKICAgIC0tbXVsdGktbWFzdGVyIHllcyBcCiAgICAtLWFwcGVuZG9ubHkgbm8gXAogICAgLS1iaW5kIDAuMC4wLjAgXAogICAgLS1wb3J0ICIkcG9ydCIgXAogICAgLS1wcm90ZWN0ZWQtbW9kZSBubyBcCiAgICAtLXNlcnZlci10aHJlYWRzIDIgXAogICAgIiR7cmVwbGljYXNbQF19Igo=

这个secret放的就是启动keydb的shell

#!/bin/bash
set -euxo pipefail

host="$(hostname)"
port="6379"
replicas=()
for node in {0..2}; do
  if [ "$host" != "my-keydb-${node}" ]; then
      replicas+=("--replicaof my-keydb-${node}.my-keydb-headless ${port}")
  fi
done
exec keydb-server /etc/keydb/redis.conf \
    --active-replica yes \
    --multi-master yes \
    --appendonly no \
    --bind 0.0.0.0 \
    --port "$port" \
    --protected-mode no \
    --server-threads 2 \
    "${replicas[@]}"
# exit

从这段解析中发现其实已经启动了主动sync的机制，并且设定replicaof设定为keydb自己的cluster主机群，那麽将这个目标主机指向redis就可以让他变成redis的备份罗。