云原生之在kubernetes环境下的持久存储配置实战

一、检查本地k8s集群环境

1.检查工作节点状态

[root@master ~]# kubectl get nodes 
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   18d   v1.24.0
node01   Ready    <none>          18d   v1.24.0
node02   Ready    <none>          18d   v1.24.0

2.检查k8s版本

[root@master ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

二、检查容器运行时

1.检查containerd状态

[root@master pvc]# systemctl status containerd.service 
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-10-21 13:19:56 CST; 39min ago
     Docs: https://containerd.io
  Process: 7745 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 7770 (containerd)
    Tasks: 109
   Memory: 203.6M
   CGroup: /system.slice/containerd.service
           ├─ 7770 /usr/bin/containerd
           ├─12576 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id e0d5d74b0ce0ce26981d0f5dd5c5c7b18b70ad89b09b1421a1565...
           ├─12577 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4304ec1b783303ed7802a4876f8a5ea26a307dbd18afdebe38bd8...
           ├─12579 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 59ace790d20e1216ee03569fc80634d31c47526281cd27774e361...
           ├─12580 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id ea0a62c3e0f80d150566a683916576b3470486467edb52216a131...
           ├─13805 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9e3bd9547b16febebc0296d6e4df25d7b5385d92a6bfc1d9556f5...
           ├─13952 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 35b43d065a9c91ce764698daaf3771510636620b3c3f44b0015d6...
           ├─18850 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 2cfcf1f889b5a0f1b3599597c4e52777b07df47655c4654933c45...
           └─18948 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id c01a9ccd059aa3864ee7629f6589d2430fade0b293d23ffc32b34...

Oct 21 13:24:10 master containerd[7770]: 2022-10-21 13:24:10.091 [INFO][21246] k8s.go 569: Cleaning up netns ContainerID=...29567"
Oct 21 13:24:10 master containerd[7770]: 2022-10-21 13:24:10.091 [INFO][21246] k8s.go 576: Releasing IP address(es) Conta...29567"
Oct 21 13:24:10 master containerd[7770]: 2022-10-21 13:24:10.091 [INFO][21246] utils.go 196: Calico CNI releasing IP addr...29567"
Oct 21 13:24:10 master containerd[7770]: 2022-10-21 13:24:10.127 [INFO][21252] ipam_plugin.go 416: Releasing address using hand...
Oct 21 13:24:10 master containerd[7770]: time="2022-10-21T13:24:10+08:00" level=info msg="About to acquire host-wide IPAM...o:357"
Oct 21 13:24:10 master containerd[7770]: time="2022-10-21T13:24:10+08:00" level=info msg="Acquired host-wide IPAM lock." ...o:372"
Oct 21 13:24:10 master containerd[7770]: 2022-10-21 13:24:10.290 [WARNING][21252] ipam_plugin.go 433: Asked to release address ...
Oct 21 13:24:10 master containerd[7770]: 2022-10-21 13:24:10.290 [INFO][21252] ipam_plugin.go 444: Releasing address using work...
Oct 21 13:24:10 master containerd[7770]: time="2022-10-21T13:24:10+08:00" level=info msg="Released host-wide IPAM lock." ...o:378"
Oct 21 13:24:10 master containerd[7770]: 2022-10-21 13:24:10.295 [INFO][21246] k8s.go 582: Teardown processing complete. ...29567"
Hint: Some lines were ellipsized, use -l to show in full.

2.检查containerd版本

[root@master pvc]# containerd -v
containerd github.com/containerd/containerd v1.6.2 de8046a5501db9e0e478e1c10cbcfb21af4c6b2d

三、安装nfs服务器

1.安装rpc跟nfs服务

yum -y install rpcbind nfs-utils

2.创建目录并修改权限

[root@master ~]# mkdir /nfs && chmod 755 /nfs

3.编辑/etc/exports 文件

[root@master ~]# cat /etc/exports
/nfs *(rw)

4.载入配置

[root@master ~]# exportfs  -rv
exporting *:/nfs

5.启动nfs服务

systemctl enable rpcbind --now  && systemctl enable nfs --now

6.检查nfs共享状态

[root@master ~]# showmount -e 192.168.3.90
Export list for 192.168.3.90:
/nfs *

四、部署storageclass

1.查看storageclass.yaml

[root@master mysql-kubernetes]# cat storageclass.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: nfs-storage
 annotations:
   storageclass.kubernetes.io/is-default-class: "true"
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
 archiveOnDelete: "true"  ## 删除pv的时候，pv的内容是否要备份

2.应用storageclass.yaml

kubectl apply -f storageclass.yaml

3.查看storageclass状态

[root@master mysql-kubernetes]# kubectl get storageclass
NAME                    PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-storage (default)   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  10m

五、部署nfs-client-provisioner

1.修改nfs-client-provisioner.yaml文件

[root@master mysql-kubernetes]# cat nfs-client-provisioner.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nfs-client-provisioner
 labels:
   app: nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
spec:
 replicas: 1
 strategy:
   type: Recreate
 selector:
   matchLabels:
     app: nfs-client-provisioner
 template:
   metadata:
     labels:
       app: nfs-client-provisioner
   spec:
     serviceAccountName: nfs-client-provisioner
     containers:
       - name: nfs-client-provisioner
         image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2
         # resources:
         #    limits:
         #      cpu: 10m
         #    requests:
         #      cpu: 10m
         volumeMounts:
           - name: nfs-client-root
             mountPath: /persistentvolumes
         env:
           - name: PROVISIONER_NAME
             value: k8s-sigs.io/nfs-subdir-external-provisioner
           - name: NFS_SERVER
             value: 192.168.3.90 ## 指定自己nfs服务器地址
           - name: NFS_PATH  
             value: /nfs/  ## nfs服务器共享的目录
     volumes:
       - name: nfs-client-root
         nfs:
           server: 192.168.3.90
           path: /nfs
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: nfs-client-provisioner-runner
rules:
 - apiGroups: [""]
   resources: ["nodes"]
   verbs: ["get", "list", "watch"]
 - apiGroups: [""]
   resources: ["persistentvolumes"]
   verbs: ["get", "list", "watch", "create", "delete"]
 - apiGroups: [""]
   resources: ["persistentvolumeclaims"]
   verbs: ["get", "list", "watch", "update"]
 - apiGroups: ["storage.k8s.io"]
   resources: ["storageclasses"]
   verbs: ["get", "list", "watch"]
 - apiGroups: [""]
   resources: ["events"]
   verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: run-nfs-client-provisioner
subjects:
 - kind: ServiceAccount
   name: nfs-client-provisioner
   # replace with namespace where provisioner is deployed
   namespace: default
roleRef:
 kind: ClusterRole
 name: nfs-client-provisioner-runner
 apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: leader-locking-nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
rules:
 - apiGroups: [""]
   resources: ["endpoints"]
   verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: leader-locking-nfs-client-provisioner
 # replace with namespace where provisioner is deployed
 namespace: default
subjects:
 - kind: ServiceAccount
   name: nfs-client-provisioner
   # replace with namespace where provisioner is deployed
   namespace: default
roleRef:
 kind: Role
 name: leader-locking-nfs-client-provisioner
 apiGroup: rbac.authorization.k8s.io

2.应用nfs-client-provisioner.yaml

[root@master mysql-kubernetes]# kubectl apply -f nfs-client-provisioner.yaml 
deployment.apps/nfs-client-provisioner created
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created

3.查看pod状态

[root@master mysql-kubernetes]#  kubectl get pods
NAME                                      READY   STATUS    RESTARTS       AGE
my-tomcat9                                1/1     Running   2 (117m ago)   18d
nfs-client-provisioner-5bbb6fc745-ncdht   1/1     Running   0              118s

六、创建pvc

1.编辑pvc.yaml

[root@master pvc]# cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
#元数据
metadata:
  name: pvc-mysql
#描述
spec:
    #访问类型
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem   #存储类型 ，块存储Block
 #存储大小
  resources:
    requests:
      storage: 8Gi

2.创建一个pvc

[root@master pvc]# kubectl apply -f pvc.yaml 
persistentvolumeclaim/pvc-mysql created

3.查看pvc状态

[root@master pvc]# kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-mysql   Bound    pvc-8ab867d2-e444-4a7e-801d-f4d04cda0164   8Gi        RWO            nfs-storage    2m23s

4.查看pv状态

[root@master pvc]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-8ab867d2-e444-4a7e-801d-f4d04cda0164   8Gi        RWO            Delete           Bound    default/pvc-mysql   nfs-storage             2m14s

七、使用pvc

1.编辑mysql.yaml文件

[root@master pvc]# cat mysql.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: mysql
  name: mysql001
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  strategy: {}
  template:
    metadata:
      labels:
        app: mysql
    spec:
      volumes:
      - name: db
        persistentVolumeClaim:
          claimName: pvc-mysql
      containers:
      - image: mysql:5.7
        name: mysql
        ports:
        - name: mysql
          containerPort: 3306
          protocol: TCP
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: root
        - name: MYSQL_DATABASE
          value: data
        resources:         
          requests:
             cpu: 0.1
             memory: 128Mi
          limits:
             cpu: 0.4
             memory:  512Mi

---
apiVersion: v1
kind: Service
#元数据
metadata:
  creationTimestamp: null
  labels:
    app: mysql
  name: mysql-svc
  #描述部分
spec:
  ports:
  - name: web-port
    port: 3306
    protocol: TCP
    targetPort: 3306
    nodePort: 30066
  selector:
    app: mysql
  type: NodePort

2.应用mysql.yaml文件

[root@master pvc]# kubectl apply -f mysql.yaml 
deployment.apps/mysql001 configured
service/mysql-svc created

3.查看pod状态

[root@master pvc]# kubectl get pod
NAME                                      READY   STATUS    RESTARTS      AGE
my-tomcat9                                1/1     Running   3 (28m ago)   18d
mysql001-595bc8cb5c-snbkj                 1/1     Running   0             65s
nfs-client-provisioner-5bbb6fc745-ncdht   1/1     Running   2 (23m ago)   11h

4.查看svc状态

[root@master pvc]# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP          18d
mysql-svc    NodePort    10.96.1.254   <none>        3306:30066/TCP   3m16s

八、访问mysql数据库

[root@docker ~]# mysql -h 192.168.3.91 -P30066 -uroot -proot
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.40 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| data               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

MySQL [(none)]>