# gitops **Repository Path**: pigful/gitops ## Basic Information - **Project Name**: gitops - **Description**: 搭建 Gitops ， 基础环境 Kubernetes + Ceph +Redis + Postgresql + Gitlab + Jenkins + Harbo - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 3 - **Forks**: 1 - **Created**: 2021-05-19 - **Last Updated**: 2024-10-12 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README 搭建 Gitops 基础环境 Kubernetes + Ceph +Redis + Postgresql + Gitlab + Jenkins + Harbor 以 Ceph 为基础构建 Gitops 环境 ## 访问 K8S 集群配置 1. hosts 文件解析： ```bash cat /etc/hosts ... 192.168.0.104 git.ik8s.com 192.168.0.104 jenkins.ik8s.com 192.168.0.104 registry.ik8s.com ``` 2. Haproxy 转发 80 和 443 端口： 查看 ingress 的端口： ```bash root@k8s04 ~# kubectl get svc -n kube-system |grep ingress traefik-ingress-service NodePort 10.68.65.207 80:33456/TCP,443:33457/TCP,8080:34265/TCP 6d20h ``` 配置 haproxy 解析到 ingress 的端口，使流量进入 k8s 集群： ```bash root@k8s04 ~# cat /etc/haproxy/haproxy.cfg ... listen https_443 bind 192.168.0.104:443 mode tcp option tcplog option dontlognull option dontlog-normal balance roundrobin server 192.168.0.101 192.168.0.104:33457 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.102 192.168.0.105:33457 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.106:33457 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.101 192.168.0.107:33457 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.102 192.168.0.108:33457 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.109:33457 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.110:33457 check inter 10s fall 2 rise 2 weight 1 listen http_80 bind 192.168.0.104:80 mode tcp option tcplog option dontlognull option dontlog-normal balance roundrobin server 192.168.0.101 192.168.0.104:33456 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.102 192.168.0.105:33456 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.106:33456 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.107:33456 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.108:33456 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.109:33456 check inter 10s fall 2 rise 2 weight 1 server 192.168.0.103 192.168.0.110:33456 check inter 10s fall 2 rise 2 weight 1 root@k8s04 ~# systemctl restart haproxy.service root@k8s04 ~# ss -ntlp |grep 80 LISTEN 0 2000 192.168.0.104:80 0.0.0.0:* users:(("haproxy",pid=52619,fd=8)) root@k8s04 ~# ss -ntlp |grep 443 LISTEN 0 2000 192.168.0.104:443 0.0.0.0:* users:(("haproxy",pid=52619,fd=7)) ``` ## 准备 Ceph 持久化存储 以 Ceph 为基础构建 Gitops 环境： ### 创建 gitops 存储池： ```bash cephadm@ceph01:~/ceph_cluster$ ceph osd pool create gitops 64 64 pool 'gitops' created cephadm@ceph01:~/ceph_cluster$ ceph osd pool application enable gitops rbd enabled application 'rbd' on pool 'gitops' cephadm@ceph01:~/ceph_cluster$ rbd pool init gitops ``` ### 创建 CephFS 文件系统 ```bash # CephFS 需要至少运行一个元数据服务器 (MDS) 守护进程 (ceph-mds) cephadm@ceph01:~/ceph_cluster$ ceph-deploy mds create ceph02 cephadm@ceph01:~/ceph_cluster$ ceph mds stat 1 up:standby # 使用 CephFS 之前需要事先于集群中创建一个文件系统，并为其分别制定元数据和数据相关的存储池 cephadm@ceph01:~/ceph_cluster$ ceph osd pool create gitopsfs-metadata 64 64 pool 'gitopsfs-metadata' created cephadm@ceph01:~/ceph_cluster$ ceph osd pool create gitopsfs-data 64 64 pool 'gitopsfs-data' created # 创建名为 gitopsfs 的文件系统，使用 cephfs-metadata 为元数据存储池，cephfs-data 为数据存储池 cephadm@ceph01:~/ceph_cluster$ ceph fs new gitopsfs gitopsfs-metadata gitopsfs-data new fs with metadata pool 13 and data pool 14 cephadm@ceph01:~/ceph_cluster$ ceph fs status gitopsfs gitopsfs - 0 clients ======== RANK STATE MDS ACTIVITY DNS INOS 0 active ceph02 Reqs: 0 /s 10 13 POOL TYPE USED AVAIL gitopsfs-metadata metadata 1536k 9.92G gitopsfs-data data 0 9.92G MDS version: ceph version 15.2.10 (27917a557cca91e4da407489bbaa64ad4352cc02) octopus (stable) ``` ### 创建 gitops 用户： ```bash cephadm@ceph01:~/ceph_cluster$ ceph auth get-or-create client.gitops mon 'allow r' mds 'allow rw' osd 'allow * pool=gitops, allow * pool=gitopsfs-data, allow * pool=gitopsfs-metadata' [client.gitops] key = AQA3BIVgF5VQOhAA45+iIC1q71UhT0yYHPZn6w== cephadm@ceph01:~/ceph_cluster$ ceph auth print-key client.gitops | base64 QVFBM0JJVmdGNVZRT2hBQTQ1K2lJQzFxNzFVaFQweVlIUFpuNnc9PQ== cephadm@ceph01:~/ceph_cluster$ ceph auth print-key client.admin | base64 QVFBa2tWbGdjSFN4Q0JBQVpoSi9PWldmZnRES1Z0elRvUDhhdHc9PQ== ``` ### k8s 访问 ceph 存储集群： #### 准备名称空间文件： （1-ns-gitops.yaml）： ```yaml --- apiVersion: v1 kind: Namespace metadata: name: gitops ``` #### 准备访问ceph用户的key： （2-ceph-users-key-secret.yaml） ```yaml # 获取 Ceph 管理员用户权限的 key # 命令：ceph auth print-key client.admin | base64 apiVersion: v1 kind: Secret metadata: name: ceph-admin-secret namespace: kube-system type: "kubernetes.io/rbd" data: key: QVFBa2tWbGdjSFN4Q0JBQVpoSi9PWldmZnRES1Z0elRvUDhhdHc9PQ== --- # 获取 Ceph gitops用户权限的 key # 命令：ceph auth print-key client.gitops | base64 apiVersion: v1 kind: Secret metadata: name: ceph-gitops-secret namespace: gitops type: "kubernetes.io/rbd" data: key: QVFBM0JJVmdGNVZRT2hBQTQ1K2lJQzFxNzFVaFQweVlIUFpuNnc9PQ== ``` #### 准备gitops项目存储卷： （3-ceph-gitops-storage.yaml）： ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gitops-data provisioner: kubernetes.io/rbd parameters: monitors: 192.168.0.131:6789,192.168.0.132:6789,192.168.0.133:6789 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: kube-system pool: gitops userId: gitops userSecretName: ceph-gitops-secret userSecretNamespace: gitops ``` 部署以上资源： ```bash root@k8s01 ~/k8s/cicd/gitops# ls 1-ns-gitops.yaml 4-cephfs-provisioner.yaml gitlab postgresql 2-ceph-users-key-secret.yaml 5-ceph-gitopsfs-storage.yaml harbor redis 3-ceph-gitops-storage.yaml 6-db-secret.yaml jenkins root@k8s01 ~/k8s/cicd/gitops# kubectl apply -f 1-ns-gitops.yaml root@k8s01 ~/k8s/cicd/gitops# kubectl apply -f 2-ceph-users-key-secret.yaml root@k8s01 ~/k8s/cicd/gitops# kubectl apply -f 3-ceph-gitops-storage.yaml ``` #### 部署 cephfs-provisioner > 这里部署了 但是没有访问成功，暂不知道原因，部署的目的只是为了动态提供一个可多节点挂载的 pvc， > > jenkins-maven-cache-pvc ，后面直接使用静态 pv 方式提供一个 pvc 使用CephFS作为持久数据卷，官方没有提供cephfs动态卷支持，使用社区提供的cephfs-provisioner： （4-cephfs-provisioner.yaml）： ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: cephfs-provisioner namespace: kube-system --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cephfs-provisioner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "delete"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cephfs-provisioner subjects: - kind: ServiceAccount name: cephfs-provisioner namespace: kube-system roleRef: kind: ClusterRole name: cephfs-provisioner apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: cephfs-provisioner namespace: kube-system rules: - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: cephfs-provisioner namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: cephfs-provisioner subjects: - kind: ServiceAccount name: cephfs-provisioner namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: name: cephfs-provisioner namespace: kube-system spec: selector: matchLabels: app: cephfs-provisioner replicas: 1 strategy: type: Recreate template: metadata: labels: app: cephfs-provisioner spec: containers: - name: cephfs-provisioner image: "registry.cn-chengdu.aliyuncs.com/ives/cephfs-provisioner:latest" env: - name: PROVISIONER_NAME value: ceph.com/cephfs command: - "/usr/local/bin/cephfs-provisioner" args: - "-id=cephfs-provisioner-1" serviceAccount: cephfs-provisioner ``` 配置StorageClass： （5-ceph-gitopsfs-storage.yaml）： ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gitopsfs-data provisioner: ceph.com/cephfs parameters: monitors: 192.168.0.131:6789,192.168.0.132:6789,192.168.0.133:6789 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: kube-system pool: gitops userId: gitops userSecretName: ceph-gitops-secret userSecretNamespace: gitops claimRoot: /gitopsfs/maven ``` 创建上面的资源： ```bash root@k8s01 ~/k8s/cicd/gitops# kubectl apply -f 4-cephfs-provisioner.yaml serviceaccount/cephfs-provisioner created clusterrole.rbac.authorization.k8s.io/cephfs-provisioner created clusterrolebinding.rbac.authorization.k8s.io/cephfs-provisioner created role.rbac.authorization.k8s.io/cephfs-provisioner created rolebinding.rbac.authorization.k8s.io/cephfs-provisioner created deployment.apps/cephfs-provisioner created root@k8s01 ~/k8s/cicd/gitops# kubectl apply -f 5-ceph-gitopsfs-storage.yaml storageclass.storage.k8s.io/gitopsfs-data created root@k8s01 ~/k8s/cicd/gitops# kubectl get pods -n kube-system -l app=cephfs-provisioner NAME READY STATUS RESTARTS AGE cephfs-provisioner-7d6d79c498-qxjqc 1/1 Running 0 81s ``` #### 部署 Jenkins maven 缓存 pvc (jenkins-maven-cache-pvc.yaml )： ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: jenkins-maven-cache-pv labels: name: jenkins-maven-cache-pv spec: capacity: storage: 2Gi accessModes: - ReadWriteMany cephfs: monitors: - 192.168.0.131:6789 - 192.168.0.132:6789 - 192.168.0.133:6789 # path: /jenkins/maven user: admin secretRef: name: ceph-admin-secret namespace: kube-system readOnly: false persistentVolumeReclaimPolicy: Retain --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-maven-cache-pvc namespace: gitops spec: selector: matchLabels: name: jenkins-maven-cache-pv accessModes: - ReadWriteMany resources: requests: storage: 2Gi volumeMode: Filesystem ``` 部署 pv/pvc ```bash root@k8s01 ~/k8s/cicd/gitops# kubectl apply -f jenkins-maven-cache-pvc.yaml root@k8s01 ~/k8s/cicd/gitops# kubectl get pvc -n gitops jenkins-maven-cache-pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE jenkins-maven-cache-pvc Bound jenkins-maven-cache-pv 2Gi RWX 6m29s ``` ## 准备 DB 密码所用的 Secret 准备 db 相关的密码，这里是 postgresql 和 gitlab 所使用（6-db-secret.yaml）： > `db_user`：加密方式：`echo -n "gitops" | base64` > > `db_pass`：加密方式：`echo -n "gitops123" | base64` ```yaml apiVersion: v1 kind: Secret metadata: name: gitops namespace: gitops data: db_user: Z2l0b3Bz db_pass: Z2l0b3BzMTIz gitlab_root_pass: Z2l0b3BzMTIz gitlab_secrets_db_key_base: bE92U1NTcHMwSDJVU2tBTS9VajhZVUZMRjhPS25xUGhwTG5ocG41N0drTQ== gitlab_secrets_otp_key_base: aVZ6Z01OUFoybjFKRk1US1ltUUVUS3lYL3VpbWpKaDBMeVhFemlmTmhVNA== gitlab_secrets_secret_key_base: VFVFNWk3SW1wT0lQSzN6cnZCTnFUU09UWjI3ZjRkTm56cVNXejF6eW5BWQ== type: Opaque ``` 创建上面的资源： ```bash root@k8s01 ~/k8s/cicd/gitops# ls 1-ns-gitops.yaml 4-cephfs-provisioner.yaml gitlab postgresql 2-ceph-users-key-secret.yaml 5-ceph-gitopsfs-storage.yaml harbor redis 3-ceph-gitops-storage.yaml 6-db-secret.yaml jenkins root@k8s01 ~/k8s/cicd/gitops# kubectl apply -f 6-db-secret.yaml ``` ## 部署 Redis ### 准备 Redis 配置清单： ```bash root@k8s01 ~/k8s/cicd/gitops/redis# ls 1-pvc-redis.yaml 2-svc-redis.yaml 3-deploy-redis.yaml 4-ingress-redis.yaml ``` #### 定义 Redis 的 pvc： （1-pvc-redis.yaml ）： ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-pvc namespace: gitops spec: storageClassName: gitops-data accessModes: - ReadWriteOnce resources: requests: storage: 2Gi ``` #### 定义 Redis 的 svc： （2-svc-redis.yaml ） ```yaml --- apiVersion: v1 kind: Service metadata: name: redis namespace: gitops labels: app: gitops-redis spec: type: ClusterIP ports: - name: redis protocol: TCP port: 6379 targetPort: redis selector: app: gitops-redis ``` #### 定义 Redis 的 deploy 控制器： （3-deploy-redis.yaml）： ```yaml --- kind: Deployment apiVersion: apps/v1 metadata: name: redis namespace: gitops labels: app: gitops-redis spec: replicas: 1 selector: matchLabels: app: gitops-redis template: metadata: name: gitops-redis labels: app: gitops-redis spec: containers: - name: gitops-redis image: 'sameersbn/redis:4.0.9-3' ports: - name: redis containerPort: 6379 protocol: TCP resources: limits: cpu: 500m memory: 1Gi requests: cpu: 200m memory: 1Gi livenessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 readinessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 volumeMounts: - name: data mountPath: /var/lib/redis volumes: - name: data persistentVolumeClaim: claimName: redis-pvc ``` #### 定义 Redis 的 Ingress： 这里可以使用 域名进行连接 redis（4-ingress-redis.yaml ）： ```yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: redis namespace: gitops annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: redis.ik8s.com http: paths: - backend: service: name: gitops-redis port: name: redis path: / pathType: ImplementationSpecific ``` ### 创建 Redis 相关资源： ```bash root@k8s01 ~/k8s/cicd/gitops/redis# kubectl apply -f . persistentvolumeclaim/redis-pvc created service/redis created deployment.apps/redis created ingress.networking.k8s.io/redis created root@k8s01 ~/k8s/cicd/gitops/redis# kubectl get pods -n gitops NAME READY STATUS RESTARTS AGE redis-7d7698b74-jcwqt 1/1 Running 1 18h root@k8s01 ~/k8s/cicd/gitops/redis# kubectl get all -n gitops NAME READY STATUS RESTARTS AGE pod/redis-7d7698b74-jcwqt 1/1 Running 1 18h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/redis ClusterIP 10.68.154.49 6379/TCP 18h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/redis 1/1 1 1 18h NAME DESIRED CURRENT READY AGE replicaset.apps/redis-7d7698b74 1 1 1 18h ``` ## 部署 Postgresql ### 准备 Postgresql 资源清单： ```bash root@k8s01 ~/k8s/cicd/gitops/postgresql# ls 1-pvc-postgresql.yaml 2-svc-postgresql.yaml 3-deploy-postgresql.yaml 4-ingress-postgresql.yaml ``` #### 定义Postgresql 的 pvc： （1-pvc-postgresql.yaml ）： ```yaml --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: postgresql-pvc namespace: gitops spec: storageClassName: gitops-data accessModes: - ReadWriteOnce resources: requests: storage: 2Gi ``` #### 定义Postgresql 的 svc： （2-svc-postgresql.yaml）： ```yaml --- kind: Service apiVersion: v1 metadata: name: pgsql namespace: gitops labels: app: postgresql spec: ports: - name: postgres protocol: TCP port: 5432 targetPort: postgres selector: app: postgresql type: ClusterIP ``` #### 定义 Postgresql 的 deploy 控制器： （3-deploy-postgresql.yaml ）： ```yaml --- kind: Deployment apiVersion: apps/v1 metadata: name: pgsql namespace: gitops labels: app: postgresql spec: replicas: 1 selector: matchLabels: app: postgresql template: metadata: name: postgresql labels: app: postgresql spec: containers: - name: postgresql image: sameersbn/postgresql:12-20200524 ports: - name: postgres containerPort: 5432 env: - name: DB_USER valueFrom: secretKeyRef: name: gitops key: db_user - name: DB_PASS valueFrom: secretKeyRef: name: gitops key: db_pass - name: DB_NAME value: 'gitlabhq_production,registry,core,notaryserver,notarysigner' - name: DB_EXTENSION value: 'pg_trgm,btree_gist' resources: requests: cpu: 200m memory: 256Mi limits: cpu: 2 memory: 2Gi livenessProbe: exec: command: ["pg_isready","-h","localhost","-U","postgres"] initialDelaySeconds: 30 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 readinessProbe: exec: command: ["pg_isready","-h","localhost","-U","postgres"] initialDelaySeconds: 5 timeoutSeconds: 1 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 volumeMounts: - name: data mountPath: /var/lib/postgresql volumes: - name: data persistentVolumeClaim: claimName: postgresql-pvc ``` #### 定义 Postgresql 的 Ingress ： 这里可以使用 域名进行连接 Postgresql（4-ingress-postgresql.yaml）： ```yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: pgsql namespace: gitops annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: pgsql.ik8s.com http: paths: - backend: service: name: pgsql port: name: postgres path: / pathType: ImplementationSpecific ``` ### 创建 Postgresql 相关资源： ```bash root@k8s01 ~/k8s/cicd/gitops/postgresql# kubectl apply -f . persistentvolumeclaim/postgresql-pvc created service/pgsql created deployment.apps/pgsql created ingress.networking.k8s.io/pgsql created root@k8s01 ~/k8s/cicd/gitops/postgresql# kubectl get pods -n gitops -l app=postgresql NAME READY STATUS RESTARTS AGE pgsql-656b98c754-7kvqc 1/1 Running 0 117s root@k8s01 ~/k8s/cicd/gitops/postgresql# kubectl get all -n gitops -l app=postgresql NAME READY STATUS RESTARTS AGE pod/pgsql-656b98c754-7kvqc 1/1 Running 0 2m21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/pgsql ClusterIP 10.68.52.64 5432/TCP 2m22s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/pgsql 1/1 1 1 2m22s NAME DESIRED CURRENT READY AGE replicaset.apps/pgsql-656b98c754 1 1 1 2m22s ``` ## 部署 Gitlab ### 准备 Gitlab 资源清单： ```bash root@k8s01 ~/k8s/cicd/gitops/gitlab# ls 1-pvc-gitlab.yaml 2-svc-gitlab.yaml 3-deploy-gitlab.yaml 4-ingress-gitlab.yaml Runner ``` #### 定义 Gitlab 的 pvc： （1-pvc-gitlab.yaml）： ```yaml --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: gitlab-pvc namespace: gitops spec: storageClassName: gitops-data accessModes: - ReadWriteOnce resources: requests: storage: 2Gi ``` #### 定义 Gitlab 的 svc： （2-svc-gitlab.yaml）： ```yaml --- kind: Service apiVersion: v1 metadata: name: gitlab namespace: gitops labels: app: gitlab spec: ports: - name: http protocol: TCP port: 80 targetPort: http - name: ssh protocol: TCP port: 22 targetPort: ssh nodePort: 31022 selector: app: gitlab type: NodePort ``` #### 定义 Gitlab 的 deploy 控制器： （3-deploy-gitlab.yaml ）： > `GITLAB_HOST` 是 自定义HTTP(S)协议Git克隆URL ```yaml --- kind: Deployment apiVersion: apps/v1 metadata: name: gitlab namespace: gitops labels: app: gitlab spec: replicas: 1 selector: matchLabels: app: gitlab template: metadata: name: gitlab labels: app: gitlab spec: containers: - name: gitlab image: 'sameersbn/gitlab:13.10.2' ports: - name: ssh containerPort: 22 - name: http containerPort: 80 - name: https containerPort: 443 env: - name: GITLAB_TIMEZONE value: Asia/Shanghai - name: GITLAB_SECRETS_OTP_KEY_BASE # Be used to encrypt 2FA secrets in the database. "long-and-random-alpha-numeric-string" valueFrom: secretKeyRef: name: gitops key: gitlab_secrets_otp_key_base - name: GITLAB_SECRETS_DB_KEY_BASE # Be used to encrypt CI secret variables, as well as import credentials, in the database. valueFrom: secretKeyRef: name: gitops key: gitlab_secrets_db_key_base - name: GITLAB_SECRETS_SECRET_KEY_BASE # Be used for password reset links, and other 'standard' auth features. valueFrom: secretKeyRef: name: gitops key: gitlab_secrets_secret_key_base - name: GITLAB_ROOT_PASSWORD valueFrom: secretKeyRef: name: gitops key: gitlab_root_pass - name: GITLAB_ROOT_EMAIL value: admin@gitops.com - name: GITLAB_HOST value: 'http://git.ik8s.com' - name: GITLAB_PORT value: '80' - name: GITLAB_SSH_PORT value: '31022' - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS value: 'true' - name: GITLAB_NOTIFY_PUSHER value: 'false' - name: DB_TYPE value: postgres - name: DB_HOST value: pgsql - name: DB_PORT value: '5432' - name: DB_USER valueFrom: secretKeyRef: name: gitops key: db_user - name: DB_PASS valueFrom: secretKeyRef: name: gitops key: db_pass - name: DB_NAME value: gitlabhq_production - name: REDIS_HOST value: redis - name: REDIS_PORT value: '6379' resources: requests: cpu: 1 memory: 1Gi limits: cpu: 2 memory: 4Gi livenessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 300 timeoutSeconds: 5 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 readinessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 30 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 volumeMounts: - name: data mountPath: /home/git/data - name: localtime mountPath: /etc/localtime volumes: - name: data persistentVolumeClaim: claimName: gitlab-pvc - name: localtime hostPath: path: /etc/localtime ``` #### 定义 Gitlab 的 Ingress : （4-ingress-gitlab.yaml）： ```yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gitlab namespace: gitops annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: git.ik8s.com http: paths: - backend: service: name: gitlab port: name: http path: / pathType: ImplementationSpecific ``` ### 创建 Gitlab 相关资源： ```bash root@k8s01 ~/k8s/cicd/gitops/gitlab# kubectl apply -f . persistentvolumeclaim/gitlab-pvc created service/gitlab created deployment.apps/gitlab created ingress.networking.k8s.io/gitlab created root@k8s01 ~/k8s/cicd/gitops/gitlab# kubectl get pods -n gitops -l app=gitlab NAME READY STATUS RESTARTS AGE gitlab-58fdbc9fc5-rft8b 1/1 Running 1 8m52s root@k8s01 ~/k8s/cicd/gitops/gitlab# kubectl get all -n gitops -l app=gitlab NAME READY STATUS RESTARTS AGE pod/gitlab-58fdbc9fc5-rft8b 1/1 Running 1 9m13s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/gitlab NodePort 10.68.158.90 80:36966/TCP,22:31022/TCP 9m13s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/gitlab 1/1 1 1 9m13s NAME DESIRED CURRENT READY AGE replicaset.apps/gitlab-58fdbc9fc5 1 1 1 9m13s ``` ### 访问 Gitlab 因为上面已经做了 域名解析 到 104 主机，这里直接在浏览器输入 git.ik8s.com， 用户名：root 密码为（4-db-secret.yaml）文件里面的 `gitlab_root_pass`：gitops123 ![image-20210426114002984](gitops.assets/image-20210426114002984.png) #### 设置 gitlab 为中文 点击头像下拉菜单 ---> preferences ---> 下拉到最后，Localization Language 下拉菜单，选择简体中文 ---> Save changes(保存) ![image-20210426114946702](gitops.assets/image-20210426114946702.png) ## 部署 Jenkins ### 准备 Jenkins 资源清单： ```bash root@k8s01 ~/k8s/cicd/gitops/jenkins# ls 1-pvc-jenkins.yaml 2-rbac-jenkins.yaml 3-svc-jenkins.yaml 4-deploy-jenkins.yaml 5-ingress-jenkins.yaml 6-ceph-gitopsfs-storage.yaml ``` #### 定义 Jenkins 的 pvc： （1-pvc-jenkins.yaml）： ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-pvc namespace: gitops spec: storageClassName: gitops-data accessModes: - ReadWriteOnce resources: requests: storage: 2Gi ``` #### 定义 Jenkins 的 rbac ： （2-rbac-jenkins.yaml）： ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-admin namespace: gitops --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: jenkins-admin rules: - apiGroups: ["extensions", "apps"] resources: ["deployments"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["services"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: jenkins-admin namespace: gitops roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins-admin subjects: - kind: ServiceAccount name: jenkins-admin namespace: gitops ``` #### 定义 Jenkins 的 svc： （3-svc-jenkins.yaml）： ```yaml apiVersion: v1 kind: Service metadata: name: jenkins namespace: gitops labels: app: jenkins spec: selector: app: jenkins type: ClusterIP ports: - name: web port: 8080 targetPort: web - name: agent port: 50000 targetPort: agent ``` #### 定义 Jenkins 的 deploy 控制器： （4-deploy-jenkins.yaml）： ```yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: gitops spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: labels: app: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccount: jenkins-admin containers: - name: jenkins #image: jenkins/jenkins:alpine image: jenkins/jenkins:2.287-jdk11 imagePullPolicy: IfNotPresent env: - name: JAVA_OPTS value: -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai ports: - containerPort: 8080 name: web protocol: TCP - containerPort: 50000 name: agent protocol: TCP resources: limits: cpu: 2 memory: 4Gi requests: cpu: 500m memory: 512Mi livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 volumeMounts: - name: jenkinshome mountPath: /var/jenkins_home securityContext: fsGroup: 1000 volumes: - name: jenkinshome persistentVolumeClaim: claimName: jenkins-pvc ``` #### 定义 Jenkins 的 Ingress : （5-ingress-jenkins.yaml）： ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: jenkins namespace: gitops spec: rules: - host: jenkins.ik8s.com http: paths: - path: / backend: service: name: jenkins port: name: web pathType: ImplementationSpecific ``` #### 定义 maven 缓存的 pvc： 这个 pvc 是为了存储 maven 构建环境的缓存，在 构建 maven 项目时，Pod 能同时挂载 cephfs 文件系统： （6-ceph-gitopsfs-storage.yaml）： ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-maven-cache-pvc namespace: gitops spec: storageClassName: gitopsfs-data accessModes: - ReadWriteMany resources: requests: storage: 2Gi ``` ### 创建 Jenkins 相关资源： ```bash root@k8s01 ~/k8s/cicd/gitops/jenkins# kubectl apply -f . persistentvolumeclaim/jenkins-pvc created persistentvolumeclaim/jenkins-maven-cache-pvc created serviceaccount/jenkins-admin created clusterrole.rbac.authorization.k8s.io/jenkins-admin created clusterrolebinding.rbac.authorization.k8s.io/jenkins-admin created service/jenkins created deployment.apps/jenkins created ingress.networking.k8s.io/jenkins created root@k8s01 ~/k8s/cicd/gitops/jenkins# kubectl get pods -n gitops -l app=jenkins NAME READY STATUS RESTARTS AGE jenkins-68b7fcc4db-cn7n8 1/1 Running 0 98s root@k8s01 ~/k8s/cicd/gitops/jenkins# kubectl get all -n gitops -l app=jenkins NAME READY STATUS RESTARTS AGE pod/jenkins-68b7fcc4db-cn7n8 1/1 Running 0 2m21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/jenkins ClusterIP 10.68.154.243 8080/TCP,50000/TCP 2m22s NAME DESIRED CURRENT READY AGE replicaset.apps/jenkins-68b7fcc4db 1 1 1 2m21s ``` ### 访问 Jenkins ![image-20210426151449349](gitops.assets/image-20210426151449349.png) 密码可以使用 `kubectl logs -n gitops jenkins-68b7fcc4db-fzzvf` 获取： ```bash root@k8s01 ~/k8s/cicd/gitops/jenkins# kubectl logs -n gitops jenkins-68b7fcc4db-cn7n8 ... Jenkins initial setup is required. An admin user has been created and a password generated. Please use the following password to proceed to installation: 98f91b9f3631465bac2d6c271dab2c10 This may also be found at: /var/jenkins_home/secrets/initialAdminPassword ... ``` 也可以使用 `kubectl exec -it -n gitops jenkins-68b7fcc4db-fzzvf -- bash` ```bash root@k8s01 ~/k8s/cicd/gitops/jenkins# kubectl exec -it -n gitops jenkins-68b7fcc4db-cn7n8 -- bash jenkins@jenkins-68b7fcc4db-fzzvf:/$ cat /var/jenkins_home/secrets/initialAdminPassword 98f91b9f3631465bac2d6c271dab2c10 ``` ![image-20210426152025648](gitops.assets/image-20210426152025648.png) 选择 安装推荐的插件 即可，安装好后创建一个 管理员用户，Jenkins URL 就是外网能访问的 Jenkins 的地址 http://jenkins.ik8s.com/ 最后部署好后即可重启，重启完成后使用刚才创建的管理员登录即可 ![image-20210426173627543](gitops.assets/image-20210426173627543.png) ## 部署 Harbor 简单的安装方法：Helm，Harbor 官方提供了对应的 Helm Chart 包，所以我们可以很容易安装。 首先下载 Harbor Chart 包到要安装的集群上： ```bash root@k8s01 ~/k8s/cicd/gitops# helm repo add bitnami https://charts.bitnami.com/bitnami root@k8s01 ~/k8s/cicd/gitops# helm pull bitnami/harbor --version 9.8.3 root@k8s01 ~/k8s/cicd/gitops# tar -xf harbor-9.8.3.tgz root@k8s01 ~/k8s/cicd/gitops# rm harbor-9.8.3.tgz root@k8s01 ~/k8s/cicd/gitops# cd harbor root@k8s01 ~/k8s/cicd/gitops/harbor# ls cert Chart.lock charts Chart.yaml ci conf README.md templates values.yaml ``` ### 准备 Harbor 资源清单： （harbor-values.yaml）： ```yaml global: storageClass: "gitops-data" internalTLS: enabled: true service: tls: commonName: registry.ik8s.com ingress: enabled: true hosts: core: registry.ik8s.com notary: notary.ik8s.com annotations: kubernetes.io/ingress.class: "traefik" traefik.ingress.kubernetes.io/redirect-entry-point: https externalURL: https://registry.ik8s.com persistence: enabled: true resourcePolicy: 'keep' persistentVolumeClaim: registry: size: 2Gi jobservice: size: 1Gi chartmuseum: size: 1Gi trivy: size: 1Gi redis: enabled: false externalRedis: host: redis.gitops.svc.cluster.local port: 6379 postgresql: enabled: false externalDatabase: host: pgsql.gitops.svc.cluster.local user: gitops password: 'gitops123' sslmode: disable coreDatabase: core notaryServerDatabase: notaryserver notaryServerUsername: gitops notaryServerPassword: gitops123 notarySignerDatabase: notarysigner notarySignerUsername: gitops notarySignerPassword: gitops123 ``` ### 创建 Harbor 相关资源： 这里使用之前创建 Redis Postgresql，上面的文档已经配置好： ```bash root@k8s01 ~/k8s/cicd/gitops/harbor# helm install harbor -f harbor-values.yaml . -n gitops NAME: harbor LAST DEPLOYED: Tue Apr 27 17:23:39 2021 NAMESPACE: gitops STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ** Please be patient while the chart is being deployed ** 1. Get the Harbor URL: You should be able to access your new Harbor installation through https://registry.ik8s.com 2. Login with the following credentials to see your Harbor application echo Username: "admin" echo Password: $(kubectl get secret --namespace gitops harbor-core-envvars -o jsonpath="{.data.HARBOR_ADMIN_PASSWORD}" | base64 --decode) ``` 等待 harbor 相关的 pod 处于 READY 状态，因为我之前部署过，镜像已经下载好，启动会很快： ```bash root@k8s01 ~/k8s/cicd/gitops/harbor# kubectl get pods -n gitops NAME READY STATUS RESTARTS AGE harbor-chartmuseum-5ff699959d-lqf4j 0/1 Running 0 36s harbor-core-58684ddc77-j4pw5 0/1 Running 0 35s harbor-jobservice-6f6fb8b9dd-4l8v8 0/1 Running 0 36s harbor-notary-server-548bc595bf-49zhk 1/1 Running 0 36s harbor-notary-signer-85f4d49b67-d42gt 1/1 Running 0 35s harbor-portal-77fcb896b7-c7258 1/1 Running 0 36s harbor-registry-598f8657d5-hblj5 0/2 Running 0 36s harbor-trivy-0 0/1 Running 0 35s ... root@k8s01 ~/k8s/cicd/gitops/harbor# kubectl get pods -n gitops NAME READY STATUS RESTARTS AGE harbor-chartmuseum-5ff699959d-lqf4j 1/1 Running 0 80s harbor-core-58684ddc77-j4pw5 1/1 Running 0 79s harbor-jobservice-6f6fb8b9dd-4l8v8 1/1 Running 0 80s harbor-notary-server-548bc595bf-49zhk 1/1 Running 0 80s harbor-notary-signer-85f4d49b67-d42gt 1/1 Running 0 79s harbor-portal-77fcb896b7-c7258 1/1 Running 0 80s harbor-registry-598f8657d5-hblj5 2/2 Running 0 80s harbor-trivy-0 1/1 Running 0 79s ... ``` ### 访问 Harbor 使用 helm 部署 harbor 时，打印了一段内容，里面有怎么获取 Harbor 的密码命令： ```bash root@k8s01 ~/k8s/cicd/gitops/harbor# kubectl get secret --namespace gitops harbor-core-envvars -o jsonpath="{.data.HARBOR_ADMIN_PASSWORD}" | base64 --decode UwYz0TSbJRroot@k8s01 ~/k8s/cicd/gitops/harbor# ``` root 是用户名，root 之前的就是密码。建议修改一下密码，因为后面要使用 docker 登录 推送镜像。这里我改为 Harbor12345 使用 浏览器访问 harbor 的域名，`harbor-values.yaml` 文件里定义的 `externalURL: https://registry.ik8s.com` ，其中 `registry.ik8s.com` 就是对外定义的域名，会提示 不是 安全的链接，点击继续前往即可： ![image-20210427175802157](gitops.assets/image-20210427175802157.png) ### 创建镜像仓库 为了后面的推送镜像至 harbor，新建一个 `spring-boot-helloworld` 镜像仓库 ![image-20210511141042542](gitops.assets/image-20210511141042542.png) ### 配置 docker 登录 Harbor 服务 #### k8s 集群外访问 harbor 直接登录 harbor 服务时，docker 会报错，需要配置 docker 信任自建的 harbor 服务： ```bash root@k8s10 ~# docker login https://registry.ik8s.com/spring-boot-helloworld Username: admin Password: Error response from daemon: Get https://registry.ik8s.com/v2/: x509: certificate signed by unknown authority ``` 因为我的 harbor 是启用的 443 端口，则使用 htpps ： ```bash root@k8s01 ~# kubectl get ingress -n gitops NAME CLASS HOSTS ADDRESS PORTS AGE gitlab git.ik8s.com 80 13d harbor-ingress registry.ik8s.com 80, 443 13d harbor-ingress-notary notary.ik8s.com 80, 443 13d jenkins jenkins.ik8s.com 80 3d pgsql pgsql.ik8s.com 80 13d redis redis.ik8s.com 80 16d root@k8s01 ~# kubectl get svc -n gitops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gitlab NodePort 10.68.169.22 80:42045/TCP,22:31022/TCP 13d harbor-chartmuseum ClusterIP 10.68.162.108 443/TCP 13d harbor-core ClusterIP 10.68.139.255 443/TCP 13d harbor-jobservice ClusterIP 10.68.44.37 443/TCP 13d harbor-notary-server ClusterIP 10.68.203.30 4443/TCP 13d harbor-notary-signer ClusterIP 10.68.236.145 7899/TCP 13d harbor-portal ClusterIP 10.68.33.252 443/TCP 13d harbor-registry ClusterIP 10.68.101.81 5443/TCP,8443/TCP 13d harbor-trivy ClusterIP 10.68.63.94 8443/TCP 13d jenkins ClusterIP 10.68.231.40 8080/TCP,50000/TCP 3d pgsql ClusterIP 10.68.55.47 5432/TCP 13d redis ClusterIP 10.68.154.49 6379/TCP 16d ``` 添加 信任地址，在 docker 的 `/etc/docker/daemon.json` 添加 `insecure-registries` 字段： ```bash root@k8s01 ~# cat /etc/docker/daemon.json { "data-root": "/var/lib/docker", "exec-opts": ["native.cgroupdriver=cgroupfs"], "registry-mirrors": [ "https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com" ], "insecure-registries": ["https://registry.ik8s.com"], "max-concurrent-downloads": 10, "live-restore": true, "log-driver": "json-file", "log-level": "warn", "log-opts": { "max-size": "50m", "max-file": "1" }, "storage-driver": "overlay2" } ``` k8s 所有节点都需要添加上面的内容，重启 docker 服务： ```bash root@k8s01 ~# systemctl restart docker ``` 再次登录 docker： ```bash root@k8s01 ~# docker login https://registry.ik8s.com/spring-boot-helloworld Authenticating with existing credentials... Stored credentials invalid or expired Username (admin): admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ``` #### k8s 集群内访问 harbor **这里使用 helm 部署的 harbor 使用单个 svc 是无法上传镜像的，仅做示例：** 集群内可以使用 svc 的地址 或 svc 的服务名进行访问 harbor 服务： ```bash root@k8s01 ~# kubectl get svc -n gitops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gitlab NodePort 10.68.169.22 80:42045/TCP,22:31022/TCP 13d harbor-chartmuseum ClusterIP 10.68.162.108 443/TCP 13d harbor-core ClusterIP 10.68.139.255 443/TCP 13d harbor-jobservice ClusterIP 10.68.44.37 443/TCP 13d harbor-notary-server ClusterIP 10.68.203.30 4443/TCP 13d harbor-notary-signer ClusterIP 10.68.236.145 7899/TCP 13d harbor-portal ClusterIP 10.68.33.252 443/TCP 13d harbor-registry ClusterIP 10.68.101.81 5443/TCP,8443/TCP 13d harbor-trivy ClusterIP 10.68.63.94 8443/TCP 13d jenkins ClusterIP 10.68.231.40 8080/TCP,50000/TCP 2d22h pgsql ClusterIP 10.68.55.47 5432/TCP 13d redis ClusterIP 10.68.154.49 6379/TCP 15d ``` 但是 svc 服务名 node 节点是无法解析的，还需要配置 dns 或指向 crondns： ```bash root@k8s01 ~# curl harbor-portal.gitops.svc.cluster.local curl: (6) Could not resolve host: harbor-portal.gitops.svc.cluster.local ``` 这里就使用最简单的，使用 svc 的 IP 地址来使用，因为我们使用 helm 部署的，会有很多的 svc，选用 harbor-portal svc 的 IP： ```bash root@k8s01 ~# docker login 10.68.33.252 Username: admin Password: Error response from daemon: Get https://10.68.33.252/v2/: x509: cannot validate certificate for 10.68.33.252 because it doesn't contain any IP SANs ``` 配置 `/etc/docker/daemon.json` 文件： ```bash root@k8s01 ~# cat /etc/docker/daemon.json { "data-root": "/var/lib/docker", "exec-opts": ["native.cgroupdriver=cgroupfs"], "registry-mirrors": [ "https://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com" ], "insecure-registries": ["registry.ik8s.com","10.68.33.252"], "max-concurrent-downloads": 10, "live-restore": true, "log-driver": "json-file", "log-level": "warn", "log-opts": { "max-size": "50m", "max-file": "1" }, "storage-driver": "overlay2" } ``` 重启 docker 服务： ```bash root@k8s01 ~# systemctl restart docker ``` 再次登录： ```bash root@k8s01 ~# docker login 10.68.33.252 Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ``` 看到登录成功了，同之前一样，k8s 的所有节点都需要修改 `/etc/docker/daemon.json` 文件，并重启 docker 服务。 # Gitops 环境的使用 ## 构建基础代码仓库 以 spring-boot-helloworld 为例： ### 创建 spring-boot-helloworld 仓库 ![image-20210426141037444](gitops.assets/image-20210426141037444.png) 创建好空白仓库后，会跳转到该仓库的操作指引： ![image-20210426141215559](gitops.assets/image-20210426141215559.png) ### 上传 spring-boot-helloworld 代码 这里引用 **马哥** 的 [spring-boot-helloworld](https://github.com/iKubernetes/spring-boot-helloworld) 做示例： ![image-20210426141425046](gitops.assets/image-20210426141425046.png) 克隆马哥的 [spring-boot-helloworld](https://github.com/iKubernetes/spring-boot-helloworld) 上传至之前搭建好的 gitlab ： ```bash root@k8s01 ~/k8s/cicd/gitops# git clone https://github.com/iKubernetes/spring-boot-helloworld.git Cloning into 'spring-boot-helloworld'... remote: Enumerating objects: 95, done. remote: Counting objects: 100% (95/95), done. remote: Compressing objects: 100% (65/65), done. remote: Total 95 (delta 22), reused 73 (delta 12), pack-reused 0 Unpacking objects: 100% (95/95), done. root@k8s01 ~/k8s/cicd/gitops# ls 1-ns-gitops.yaml 3-ceph-gitops-storage.yaml gitlab jenkins redis 2-ceph-users-key-secret.yaml 4-db-secret.yaml harbor postgresql spring-boot-helloworld root@k8s01 ~/k8s/cicd/gitops# cd spring-boot-helloworld/ root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# rm .git -rf root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git config --global user.name "Administrator" root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git config --global user.email "admin@gitops.com" root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git init Initialized empty Git repository in /root/k8s/cicd/gitops/spring-boot-helloworld/.git/ root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git remote add origin http://git.ik8s.com/root/spring-boot-helloworld.git root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git add . root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git commit -m "Initial commit" [master (root-commit) 9fcf75a] Initial commit 15 files changed, 534 insertions(+) create mode 100644 .gitignore create mode 100644 Dockerfile create mode 100644 Jenkinsfile create mode 100644 LICENSE create mode 100644 README.md create mode 100644 deploy/01-namespace.yaml create mode 100644 deploy/02-service.yaml create mode 100644 deploy/03-deployment.yaml create mode 100644 pom.xml create mode 100644 src/main/java/com/neo/Application.java create mode 100644 src/main/java/com/neo/controller/HelloWorldController.java create mode 100644 src/main/resources/application.properties create mode 100644 src/test/java/com/neo/ApplicationTests.java create mode 100644 src/test/java/com/neo/controller/HelloTests.java create mode 100644 src/test/java/com/neo/controller/HelloWorldControlerTests.java root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git checkout -b develop Switched to a new branch 'develop' root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git push --all origin Username for 'http://git.ik8s.com': root Password for 'http://root@git.ik8s.com': Counting objects: 30, done. Delta compression using up to 4 threads. Compressing objects: 100% (23/23), done. Writing objects: 100% (30/30), 8.41 KiB | 1.40 MiB/s, done. Total 30 (delta 2), reused 0 (delta 0) remote: remote: To create a merge request for develop, visit: remote: http://localhost/root/spring-boot-helloworld/-/merge_requests/new?merge_request%5Bsource_branch%5D=develop remote: To http://git.ik8s.com/root/spring-boot-helloworld.git * [new branch] develop -> develop * [new branch] master -> master ``` 返回 gitlab 网页刷新一下： ![image-20210426142858454](gitops.assets/image-20210426142858454.png) ## 完善 Jenkins ### 使用国内镜像源 点击 页面下的 **Jenkins中文社区** ![image-20210426162407710](gitops.assets/image-20210426162407710.png) 复制 页面中 中心地址，点击 设置更新中心地址，修改为刚复制的 地址，更新提交后在进入 点击 使用 ![image-20210426162500927](gitops.assets/image-20210426162500927.png) 如果 上面写的地址无法访问，可以使用 清华大学的：https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json ![image-20210426162835212](gitops.assets/image-20210426162835212.png) ### 测试 Jenkins 使用 新建任务： ![image-20210428103159538](gitops.assets/image-20210428103159538.png) 如果发生错误，请看 **Jenkins 报错处理** ，上面点击确定后到达如下界面： ![image-20210428104056222](gitops.assets/image-20210428104056222.png) 页面拉到最后，在 **流水线** 代码编辑区域直接编辑一下内容： ```groovy pipeline{ agent any stages { stage('Hello') { steps { echo "Hello pipeline" } } } } ``` ![image-20210428104201479](gitops.assets/image-20210428104201479.png) 点击保存后，会跳转到这个任务的页面，点击左边栏的 **立即构建** 就会立即执行这个任务，如果没有语法错误就会看到下面的内容： ![image-20210428104922231](gitops.assets/image-20210428104922231.png) 点击 **Logs** 能跳出 执行的日志信息。 ### Jenkins 报错处理 #### 403 No valid crumb 错误内容： ![image-20210428103301966](gitops.assets/image-20210428103301966.png) 解决办法： 系统管理 ---> 全局安全配置 ---> 跨站请求伪造保护 ---> Crumb Issuer 项 勾选 启用代理兼容 ![image-20210428103610158](gitops.assets/image-20210428103610158.png) ### 安装 kubernetes 插件 安装 **kubernetes plugin**， 点击 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin 勾选安装即可。 ![image-20210426162101053](gitops.assets/image-20210426162101053.png) 勾选安装完成后重启 Jenkins ### 访问 K8S 集群配置 安装完毕后，点击 Manage Jenkins(系统管理) —> Configure System(系统配置) —> (拖到最下方Cloud [a separate configuration page](configureClouds)) —> Add a new cloud —> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息。 **Kubernetes Cloud details** ![image-20210510104229901](gitops.assets/image-20210510104229901.png) **Pod Templates** ![image-20210508101504081](gitops.assets/image-20210508101504081.png) #### 测试使用 jenkins slave ![image-20210508101846995](gitops.assets/image-20210508101846995.png) ![image-20210508102230077](gitops.assets/image-20210508102230077.png) ```groovy pipeline { agent { kubernetes { inheritFrom 'jenkins-slave' } } stages { stage ("Hello") { steps { sh 'java -version' } } } } ``` 保存后点击**立即构建**，查看控制台输出： ![image-20210510104602400](gitops.assets/image-20210510104602400.png) 可以看到最后执行成功了。以上基本可以使用了，下面开始部署 maven 工程。 ## 构建 maven 项目 ### 定义 Pod 模板 **系统管理** **节点管理** 找到 **Configure Clouds** ，点击 **Pod Templates** ，添加一个 Pod 模板 ![image-20210510145018780](gitops.assets/image-20210510145018780.png) 继续添加一个 PVC 卷，使用之前定义的 maven 缓存的那个 pvc ，挂载的路径需与下图一致。 ```bash root@k8s01 ~# kubectl get pvc -n gitops |grep maven jenkins-maven-cache-pvc Bound jenkins-maven-cache-pv 2Gi RWX 2d18h ``` ![image-20210510112713260](gitops.assets/image-20210510112713260.png) 保存好后，进行测试。 ### 测试 maven 项目 新建一个 **pipeline-demo-03** **流水线** 任务，使用以下脚本进行构建。 ```groovy pipeline { agent { kubernetes { inheritFrom 'maven-3.6' } } stages { stage ("Hello") { steps { container('maven') { sh 'mvn -version' } } } } } ``` ![image-20210510113901514](gitops.assets/image-20210510113901514.png) 保存完成后点击立即构建，查看 Console Output 信息 ![image-20210510145635337](gitops.assets/image-20210510145635337.png) 可以看到构建的项目已经执行成功。 ## 部署 spring-boot-helloworld ### 创建任务 新建一个名为 spring-boot-helloworld 的 流水线任务，并写入下面的代码： ```groovy pipeline { environment { appName = "spring-boot-helloworld" appVersion = "v0.9.0" } agent { kubernetes { inheritFrom 'maven-3.6' // 指明所用的模板 } } stages { stage('Source') { steps { git branch: 'develop', url: 'http://gitlab.gitops.svc.cluster.local/root/spring-boot-helloworld.git' // 克隆代码，原本gitlab地址是 git.ik8s.com/root/spring-boot-helloworld.git，使用上面的访问较快 } } stage('Build') { steps { container('maven') { sh 'mvn clean test package' // 清理代码树，执行构建，调 test 做代码测试，再次打包。 } } } } } ``` ![image-20210510151806798](gitops.assets/image-20210510151806798.png) 最后保存，跳转页面执行 立即构建，查看控制台输出信息，可以看到 maven 会下载很多插件。 ![image-20210510152251345](gitops.assets/image-20210510152251345.png) 这里等它下载，说明我们的构建应该没什么问题了 ![image-20210510152949475](gitops.assets/image-20210510152949475.png)最后看到 SUCCESS ### 修改Pod模板添加docker 添加 docker 容器，使得任务支持镜像制作： ![image-20210510153856364](gitops.assets/image-20210510153856364.png) ![image-20210510154037567](gitops.assets/image-20210510154037567.png) 点击 应用 并 保存。 ### 添加镜像制作步骤 修改 spring-boot-helloworld 流水线，新增镜像制作代码段： ```groovy pipeline { environment { appName = "spring-boot-helloworld" appVersion = "v0.9.0" } agent { kubernetes { inheritFrom 'maven-and-docker' } } stages { stage('Source') { steps { git branch: 'develop', url: 'http://gitlab.gitops.svc.cluster.local/root/spring-boot-helloworld.git' } } stage('Build') { steps { container('maven') { sh 'mvn clean test package' } } } stage('Building app image') { steps { container('docker') { script { dockerImage = docker.build appName + ":" + appVersion } } } } } } ``` ![image-20210510155625741](gitops.assets/image-20210510155625741.png) 更新代码后，点击 应用 并且 保存。因为上面用到了 docker build 命令，如果现在执行构建会出错。所以需要 jenkins 安装 docker 的插件。 ### 安装 docker 插件 在 jenkins 插件管理里面安装 docker 相关插件： ![image-20210510155819143](gitops.assets/image-20210510155819143.png) 等安装好重启 jenkins 后即可构建 spring-boot-helloworld 任务 ### 执行任务构建docker镜像 再次执行 spring-boot-helloworld ，因为之前添加了 docker build 代码，将会看到镜像制作的过程： ![image-20210510170216355](gitops.assets/image-20210510170216355.png) 在对应的节点上会有一个 spring-boot-helloworld 的镜像： ```bash # 查看 任务执行 pod 所在的节点 root@k8s01 ~/k8s/cicd/gitops# kubectl get pods -n gitops -o wide | grep spring spring-boot-helloworld-8-qgz1f-bsxx4-dn2cq 3/3 Terminating 0 101s 172.20.5.204 192.168.0.109 # 查看节点上的 docker image root@k8s09 ~# docker image ls |grep spring spring-boot-helloworld v0.9.0 59574454995e 4 minutes ago 165MB ``` 测试运行以下制作的镜像，看能否正常运行： ``` root@k8s09 ~# docker run --name test --rm spring-boot-helloworld:v0.9.0 . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.1.3.RELEASE) ....... ``` 打开新终端，查看 test 容器的 IP 地址： ```bash root@k8s09 ~# docker inspect test |grep IPAddress "SecondaryIPAddresses": null, "IPAddress": "172.17.0.2", "IPAddress": "172.17.0.2", ``` 使用 `curl` 命令访问是否能提供服务： ```bash root@k8s09 ~# curl 172.17.0.2 Hello Spring Boot 2.0!root@k8s09 ~# root@k8s09 ~# curl 172.17.0.2/hello Hello Worldroot@k8s09 ~# ``` ### jenkins 配置登录 harbor 服务 上传镜像是需要登录镜像服务器的，这里需要有账号和密码： * jenkins 提供 凭据 功能提供账号和密码，在 pipeline 语法中可调用。 * 另外就是使用 `docker login -p xx -u xx` 命令直接指定用户和密码，这样容易泄露用户名和密码 所有这里选择添加 jenkins 凭据： ![image-20210511151756916](gitops.assets/image-20210511151756916.png) ![image-20210511151828813](gitops.assets/image-20210511151828813.png) ![image-20210511152218358](gitops.assets/image-20210511152218358.png) 至此，一个 凭据及添加完成，在 jenkins 任务里面可以调用了，调用时写 添加的 ID 名称。 ### 添加上传镜像步骤 修改 spring-boot-helloworld 的 流水线脚本，加上上传镜像的代码： 因为 harbor 服务里面的项目不是指一个 REPOSITORY，所有 registry 的值应该是 `域名/项目名` ```groovy pipeline { environment { appName = "spring-boot-helloworld" appVersion = "v0.9.0" registryCredential = "harboradmin" // 调用的 凭据ID registry = "registry.ik8s.com/spring-boot-helloworld" // harbor 的地址 dockerImage = "" } agent { kubernetes { inheritFrom 'maven-and-docker' } } stages { stage('Source') { steps { git branch: 'develop', url: 'http://gitlab.gitops.svc.cluster.local/root/spring-boot-helloworld.git' } } stage('Build') { steps { container('maven') { sh 'mvn clean test package' } } } stage('Building app image') { steps { container('docker') { script { dockerImage = docker.build appName + ":" + appVersion } } } } stage('Push app image') { steps { container('docker') { script { docker.withRegistry( registry, registryCredential ) { dockerImage.push() } } } } } } } ``` ![image-20210511165502676](gitops.assets/image-20210511165502676.png) 编辑好后应用并保存，执行构建： 这里出现了问题： ```bash Executing sh script inside container docker of pod spring-boot-helloworld-13-dhxhw-sk2px-q5gwq Executing command: "docker" "login" "-u" "admin" "-p" ******** "https://registry.ik8s.com" exit WARNING! Using --password via the CLI is insecure. Use --password-stdin. Error response from daemon: Get https://registry.ik8s.com/v2/: unauthorized: authentication required ``` 上传镜像的步骤无论怎么改都是失败，都是卡在登录那一步，还使用了下面的代码也不行： ```groovy stage('Push app image') { steps { withCredentials([usernamePassword(credentialsId: 'harboradmin', passwordVariable: 'harboradminPassword', usernameVariable: 'harboradminUser')]) { sh "docker login -u ${harboradminUser} -p ${harboradminPassword} ${registry}" sh "docker push ${registry}/${appName}:${appVersion}" } } } ``` 只要密码是变量就提示上面的错误，到最后我就使用了明文密码，才成功： ```groovy stage('Push app image') { steps { container('docker') { sh "docker login -u admin -p Harbor12345 ${registry}" sh "docker push ${registry}/${appName}:${appVersion}" } } } ``` 完整代码如下： ```groovy pipeline { environment { appName = "spring-boot-helloworld" appVersion = "v0.9.0" registryCredential = "harboradmin" registry = "registry.ik8s.com/spring-boot-helloworld" dockerImage = "" } agent { kubernetes { inheritFrom 'maven-and-docker' } } stages { stage('Source') { steps { git branch: 'develop', url: 'http://gitlab.gitops.svc.cluster.local/root/spring-boot-helloworld.git' } } stage('Build') { steps { container('maven') { sh 'mvn clean test package' } } } stage('Building app image') { steps { container('docker') { script { dockerImage = docker.build registry + "/" + appName + ":" + appVersion } } } } stage('Push app image') { steps { container('docker') { sh "docker login -u admin -p Harbor12345 ${registry}" sh "docker push ${registry}/${appName}:${appVersion}" } } } } } ``` 最后也是构建了几十次才成功： ![image-20210511200841756](gitops.assets/image-20210511200841756.png) 上传成功后在 harbor 上可以看到镜像： ![image-20210512102605195](gitops.assets/image-20210512102605195.png) ## CI 流水线 Gitlab 通知 Jenkins 自动触发项目构建 ### 配置Gitlab允许外发 以 Gitlab 管理员的身份，设置系统在外发请求中，允许 Webhook 和服务对本地网络的请求； ![image-20210512103600197](gitops.assets/image-20210512103600197.png) ### 为Gitlab用户添加SSH公钥 访问 Gitlab 项目时提示 **添加SSH密钥** ![image-20210512144111523](gitops.assets/image-20210512144111523.png) 点击**添加SSH密钥**，添加密钥 ![image-20210512144216666](gitops.assets/image-20210512144216666.png) **生成密钥** 生成 SSH 密钥对，将公钥保存于 Gitlab 用户账号上，以使得 git 或 jenkins 等客户端使用匹配的私钥认证到 Gitlab 服务上： 在 Jenkins Server 上以 jenkins 用户的身份生成： ```bash root@k8s01 ~# adduser --disabled-password --gecos "" jenkins root@k8s01 ~# su - jenkins jenkins@k8s01:~$ ssh-keygen -t rsa -P '' jenkins@k8s01:~$ ls .ssh/ id_rsa id_rsa.pub jenkins@k8s01:~$ cat .ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTceH9QRyOMRxceyewn/DSs4Ij72MyUCYUZZalddnDzBVnmTsmelRogQJTIYOiRc2DQ8m2SzYpY+gnYDbGGbQVRiksrBTmoyDFatL7TIEPk2sHNtjzNYF5aAZWR3wXd28ngloJ3p4LGV5zZUNUsUEcX9RygtKFRShpGdmv2Ek+7KfPG0a8xiLKhapY5UH3IE8gHayErqUoodLVZrXrXdNjaRlplRkseZGUJgecrjart4abdP4rYN110P8TP2c7znghxqHleo5atlQqhvdyMLe5lBd7jyaswn3xXkmJxn4KW5Ulx/YqfTKVsqHzsI1oTgdhTlguBIv4nAkEPeIBSFQF jenkins@k8s01.ik8s.com ``` 把上面的密钥复制到 Gitlab 添加密钥的输入框里面： ![image-20210512144523875](gitops.assets/image-20210512144523875.png) ### 在Jenkins添加SSH私钥 在 Jenkins 上通过凭据添加 SSH 私钥以认证到 Gitlab，复制 SSH 密钥对的私钥信息，保存为 Jenkins 上的凭据： ```bash jenkins@k8s01:~$ cat .ssh/id_rsa -----BEGIN RSA PRIVATE KEY----- MIIEowIBAAKCAQEA03Hh/UEcjjEcXHsnsJ/w0rOCI+9jMlAmFGWWpXXZw8wVZ5k7 JnpUaIECUyGDokXNg0PJtks2KWPoJ2A2xhm0FUYpLKwU5qMgxWrS+0yBD5NrBzbY 8zWBeWgGVkd8F3dvJ4JaCd6eCxlec2VDVLFBHF/UcoLShUUoaRnZr9hJPuynzxtG vMYiyoWqWOVB9yBPIB2shK6lKKHS1Wa1613TY2kZaZUZLHmRlCYHnK42q7eGm3T+ K2DdddD/Ez9nO854Icah5XqOWrZUKob3cjC3uZQXe48mrMJ98V5JicZ+CluVJcf2 Kn0ylbKh87CNaE4HYU5YLgSL+JwJBD3iAUhUBQIDAQABAoIBABC90pvfvOgRQWKd Ka5Va5bzOZmoyZzhNbKlvIDLTPuwu/0g58sLIoXHuSxl2etORZBgHKd/WwF59Jax ioSy4kxQ6s+rrFf6kIyxAZV+IDXggTfoAtN26BJ5xw6ryEp/XWCudHyVkMPmNVFD skoRYsthVwveCdcde66eqlq+8EW4CE1HvJK3kG4OPlxuicACkSMYhJlg45cZ3c+r iJOak+Ssn2YjU4YxSDQUjE/xhobk2Uur3E+onMXeie5nX4G7LBS71dS4B1u2afeU INGAnH+OgbChZPfDZygF6DDvEwG2kDlZaO69qbZLEUOy2hPLcXWA+vJ4X4Y6GDrN 73rR0h0CgYEA8qe9mMqFJ4pk+YfZeh1r0yRWxoUZsOmK1oaX+O/EYDLc3Q5GDQ1A tDc78AT4se0jh+A8PMM5UwPLgfP+27atSO+imxbkXzBWRPcsHtB9dympJDTkkkl2 diZ7DqGexbXw6MaVN/GU9NGWZqvY6YtMxKqGEnTpReW9u9Rb6cPo4YcCgYEA3xK9 8Ia+7UIRYFFShK2iEzZa0fkxi5tsMRHpOBEzWd5H2olzX+qwlN8r4sefGOCNLW0R MIiFpqQuDVTdSYEmOc68DLMpyojc/eJkJqYR4H+WbVMLl6LfJtpG7QO4bEP1wI3j Js4qedvAr/hCUE6ZGgPfF58wlja0WBhXeNG1cRMCgYAQIsNi06TDGlrYlQYLLsWq xrHWhadNsln/JfgZH6iJVRiGwpo55/WzhjDAJzR6cHB6apjW2YyITqpLu/PonF8t iHHhqkYJZmd2MpGGgwq0z9plg/bnG2d7N9fbAzhRoWWhtQLbM39aQE2mCitkbFTw hv5fX9LbSQmy4c4y30ovJQKBgDhaV7lnkNwHELSwVpRF+Oe5l3/r3+RQwygySiRr 0/kj+irvBkJ421sAdem4XCzArWmIYAtOsdTDLQ8ZHT3wbmO0IjVjHW+Y25sXkg1h bqq0EKBVllwcPRtnkPtXMUPId1DJh1TKSFi4dxj9MmNvN6YmzDj/chrAfxayL4bK E3gZAoGBAKz1DdFXnCoeYQ7HEP/SFrodDqstihqdFZoGzUOr/tzgATP/apmFQByC dw0qP3zqaN9TjfIFZJGu68OOoaO4MTc6oP6LCzPISJR137CwZYFoFaXtKgt9vkot s9CnOjpwTFH1Be8mBZfESD5x+guhPTswM+dL1zElp3pDJl7cauqS -----END RSA PRIVATE KEY----- ``` 复制上面的加密信息到 jenkins 添加凭据里面： ![image-20210512150719685](gitops.assets/image-20210512150719685.png) ### Gitlab 创建 Access Token User Account --> Settings --> Access Token ![image-20210512151218135](gitops.assets/image-20210512151218135.png) 点击创建后会生成一个 token，注意：离开这个页面或刷新，将找不到这个 token，最好保存好： ![image-20210512151311951](gitops.assets/image-20210512151311951.png) 我生成的 token ：Yve58sSNejG5Cei3_t1G ### Jenkins 安装 Gitlab 插件 ![image-20210512151604328](gitops.assets/image-20210512151604328.png) ![image-20210512151614990](gitops.assets/image-20210512151614990.png) 点击安装，勾选重启 jenkins ，后等待安装重启完成。 ### Jenkins 添加 Gitlab Access Token jenkins 安装完 gitlab 插件后，添加 凭据 的 类型 会多一个 Gitlab API token 选项，选此类型添加一个 凭据： ![image-20210512152520044](gitops.assets/image-20210512152520044.png) ### Jenkins 授权启用 project 端点 在 Jenkins 上授权启用 /project 端点以创建 Gitlab 连接，连接 gitlab 需要填写地址，因为这里都是在 k8s 集群内，就使用 svc 进行访问，注意填写 svc 的端口： ```bash root@k8s01 ~# kubectl get svc -n gitops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gitlab NodePort 10.68.169.22 80:42045/TCP,22:31022/TCP 14d harbor-chartmuseum ClusterIP 10.68.162.108 443/TCP 14d harbor-core ClusterIP 10.68.139.255 443/TCP 14d harbor-jobservice ClusterIP 10.68.44.37 443/TCP 14d harbor-notary-server ClusterIP 10.68.203.30 4443/TCP 14d harbor-notary-signer ClusterIP 10.68.236.145 7899/TCP 14d harbor-portal ClusterIP 10.68.33.252 443/TCP 14d harbor-registry ClusterIP 10.68.101.81 5443/TCP,8443/TCP 14d harbor-trivy ClusterIP 10.68.63.94 8443/TCP 14d jenkins ClusterIP 10.68.231.40 8080/TCP,50000/TCP 3d22h pgsql ClusterIP 10.68.55.47 5432/TCP 14d redis ClusterIP 10.68.154.49 6379/TCP 16d ``` ![image-20210512154442982](gitops.assets/image-20210512154442982.png) 填写好信息后，点击 **Test Connection** ，提示 **Success** 则表示 jenkins 通过令牌连通 gitlab，然后 应用 并保存。 ### 配置 Jenkins 经由 Gitlab 触发 配置 jenkins 项目可经由 Gitlab 上的事件触发： * 在 Jenkins 的 pipeline 和 freestyle 类型的项目上，选择使用 Gitlab connection * 接着选中 “Build when a change is pushed to GitLab”，并按需勾选允许的触发事件 * 最后 配置如何通知给 Gitlab 配置 jenkins spring-boot-helloworld 项目： ![image-20210512155816664](gitops.assets/image-20210512155816664.png) 考虑到安全性问题，我们还需要 对构建进行控制，不是谁上传代码都执行构建： 上面页面下拉，点击 **高级** 配置，找到 **Secret token** 点击 **Generate** 创建一个 Secret token，注意另外保存这串密文： ![image-20210512160604206](gitops.assets/image-20210512160604206.png) > GitLab webhook URL: http://jenkins.ik8s.com/project/spring-boot-helloworld > > Secret token: af8d804e44a3947a8b69a2688a34ac0d ### 配置 Gitlab Webhooks 去 Gitlab 页面找到 spring-boot-helloworld 项目，填写 jenkins 地址和 Secret token： jenkins 地址就是 配置 构建触发器 时 显示的那个地址，这里可以改为集群内的 svc 地址，注意 svc 的端口。 >GitLab webhook URL: http://jenkins.gitops.svc.cluster.local/project/spring-boot-helloworld 设置 --> Webhooks ： ```bash root@k8s01 ~# kubectl get svc -n gitops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... jenkins ClusterIP 10.68.231.40 8080/TCP,50000/TCP 3d22h ... ``` ![image-20210512164704075](gitops.assets/image-20210512164704075.png) ![image-20210512164222406](gitops.assets/image-20210512164222406.png) 至此 Gitlab 上面的 spring-boot-helloworld 即可推送给 jenkins 的 spring-boot-helloworld 项目了，上面页面添加完成后，页面下方有一个测试操作，可以点击测试，选择一个 推送事件，看看 jenkins 是否执行构建。 ![image-20210512164517440](gitops.assets/image-20210512164517440.png) 看 jenkins 的 spring-boot-helloworld 项目构建信息，可以看到是 “Started by GitLab push by Administrator”： ![image-20210512165239539](gitops.assets/image-20210512165239539.png) 至此，当有人上传代码到 Gitlab 服务的 spring-boot-helloworld 项目，会触发 jenkins 服务的 spring-boot-helloworld 项目的构建。 其他问题，至目前推送的镜像的版本会被顶替掉，可以把 镜像 tag 设置为变量： ```bash root@k8s10 ~# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE registry.ik8s.com/spring-boot-helloworld/spring-boot-helloworld v0.9.0 430addbd1b5d 3 minutes ago 165MB registry.ik8s.com/spring-boot-helloworld/spring-boot-helloworld 69f81806caf4 21 hours ago 165MB ... ``` ![image-20210512170039662](gitops.assets/image-20210512170039662.png) ### 使用Jenkinsfile进行构建 通过之前的操作发现，修改Jenkins构建代码比较麻烦，如修改打镜像的tag标签，无法做到最好体验，可以使用 Jenkinsfile 文件存储构建脚本，配置 jenkins 构建时使用该文件进行构建： * 修改 spring-boot-helloworld 代码的内容，把 jenkins 服务里的 spring-boot-helloworld 项目的 pipeline 代码 复制到 Jenkinsfile 文件里面，修改一下 appVersion 版本，避免镜像版本号被顶掉： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# ls deploy Dockerfile Jenkinsfile LICENSE pom.xml README.md src root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# cat Jenkinsfile ``` ```groovy pipeline { environment { appName = "spring-boot-helloworld" appVersion = "v0.9.1" registryCredential = "harboradmin" registry = "registry.ik8s.com/spring-boot-helloworld" dockerImage = "" } agent { kubernetes { inheritFrom 'maven-and-docker' } } stages { stage('Source') { steps { git branch: 'develop', url: 'http://gitlab.gitops.svc.cluster.local/root/spring-boot-helloworld.git' } } stage('Build') { steps { container('maven') { sh 'mvn clean test package' } } } stage('Building app image') { steps { container('docker') { script { dockerImage = docker.build registry + "/" + appName + ":" + appVersion } } } } stage('Push app image') { steps { container('docker') { sh "docker login -u admin -p Harbor12345 ${registry}" sh "docker push ${registry}/${appName}:${appVersion}" } } } } } ``` * 修改 pom.xml " 0.9.0-SNAPSHOT " 为"0.9.1-SNAPSHOT"： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep SNAPSHOT pom.xml 0.9.0-SNAPSHOT root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# sed -i 's/0.9.0-SNAPSHOT/0.9.1-SNAPSHOT/' pom.xml root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep SNAPSHOT pom.xml 0.9.1-SNAPSHOT ``` * 修改 ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep 0.9.0 src/main/java/com/neo/controller/HelloWorldController.java return "version 0.9.0"; root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# sed -i 's/0.9.0/0.9.1/' src/main/java/com/neo/controller/HelloWorldController.java root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep 0.9. src/main/java/com/neo/controller/HelloWorldController.java return "version 0.9.1"; ``` 重新配置 jenkins 的构建配置： ![image-20210514174836175](gitops.assets/image-20210514174836175.png) 注意 Jenkinsfile 文件一定要在 spring-boot-helloworld 的 根目录下，否则上面的配置会找不到脚本执行构建任务： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# ls deploy Dockerfile Jenkinsfile LICENSE pom.xml README.md src ``` 上传代码至 Gitlab 的 spring-boot-helloworld 项目： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# ls deploy Dockerfile Jenkinsfile LICENSE pom.xml README.md src root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git add . root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git commit -m "update to v0.9.2" [develop 4a49447] update to v0.9.2 3 files changed, 47 insertions(+), 71 deletions(-) rewrite Jenkinsfile (62%) root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git push origin develop Username for 'http://git.ik8s.com': root Password for 'http://root@git.ik8s.com': Counting objects: 11, done. Delta compression using up to 4 threads. Compressing objects: 100% (9/9), done. Writing objects: 100% (11/11), 1022 bytes | 1022.00 KiB/s, done. Total 11 (delta 4), reused 0 (delta 0) remote: remote: To create a merge request for develop, visit: remote: http://localhost/root/spring-boot-helloworld/-/merge_requests/new?merge_request%5Bsource_branch%5D=develo remote: To http://git.ik8s.com/root/spring-boot-helloworld.git 9fcf75a..4a49447 develop -> develop ``` 可以看到 jenkins 被触发构建： ![image-20210514175307184](gitops.assets/image-20210514175307184.png) 可以在 镜像仓库看到自动上传的镜像及版本： ![image-20210514181831909](gitops.assets/image-20210514181831909.png) 至此，CI 流水线已经自动完成。 ## CD 流水线 为了安全起见，应该要有一个独立的流水线，纯用于CD流水线的 jenkins，是不应该有 gitlab 和镜像仓库具有写操作的凭据，这样能防止部署操作导致代码和镜像被替换之类的结果。 ### 创建部署专用仓库 这里为了简单，使用同一个 jenkins ，新建一个 spring-boot-helloworld-deployment 代码仓库： ![image-20210514183746231](gitops.assets/image-20210514183746231.png) ### 准备部署资源清单 先克隆 spring-boot-helloworld-deployment 项目到本地： ```bash root@k8s01 ~/k8s/cicd/gitops# git clone http://git.ik8s.com/root/spring-boot-helloworld-deployment.git Cloning into 'spring-boot-helloworld-deployment'... remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (3/3), done. ``` 准备一个 专用于 kubernetes 清单的目录： ```bash root@k8s01 ~/k8s/cicd/gitops# cd spring-boot-helloworld-deployment/ root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# mkdir kubernetes root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# ls kubernetes README.md root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# cd kubernetes/ ``` 创建一个 名称空间 类型的资源清单，（1-ns.yaml）： ```yaml kind: Namespace apiVersion: v1 metadata: name: helloworld ``` 创建一个 service 类型的资源清单，（2-svc.yaml）： ```yaml kind: Service apiVersion: v1 metadata: name: helloworld namespace: helloworld labels: app: spring-boot-helloworld spec: type: ClusterIP ports: - name: http protocol: TCP port: 80 targetPort: http selector: app: spring-boot-helloworld ``` 如果需要使用域名访问的话，需要创建一个 ingress 类型的资源清单（3-ingress.yaml）： ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: helloworld namespace: helloworld spec: rules: - host: helloworld.ik8s.com http: paths: - path: / backend: service: name: helloworld port: name: http pathType: ImplementationSpecific ``` 最后准备 deployment 类型的资源清单（4-deployment.yaml）： > 注意更换 image 的地址，换成你自己镜像仓库的地址，镜像名称以及版本 ```yaml kind: Deployment apiVersion: apps/v1 metadata: name: helloworld namespace: helloworld labels: app: spring-boot-helloworld spec: replicas: 4 selector: matchLabels: app: spring-boot-helloworld template: metadata: name: spring-boot-helloworld labels: app: spring-boot-helloworld spec: containers: - name: spring-boot-helloworld image: 'registry.ik8s.com/spring-boot-helloworld/spring-boot-helloworld:v0.9.1' ports: - name: http containerPort: 80 protocol: TCP ``` ### 上传资源清单文件 把准备好的资源清单文件上传至 spring-boot-helloworld-deployment 代码仓库，并创建一个 develop 分支： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment/kubernetes# git add . root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment/kubernetes# git commit -m "Init Commit" [master 5f11ced] Init Commit 4 files changed, 62 insertions(+) create mode 100644 kubernetes/1-ns.yaml create mode 100644 kubernetes/2-svc.yaml create mode 100644 kubernetes/3-ingress.yaml create mode 100644 kubernetes/4-deployment.yaml root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment/kubernetes# git checkout -b develop Switched to a new branch 'develop' root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment/kubernetes# git push --all origin Username for 'http://git.ik8s.com': root Password for 'http://root@git.ik8s.com': Counting objects: 7, done. Delta compression using up to 4 threads. Compressing objects: 100% (7/7), done. Writing objects: 100% (7/7), 1.05 KiB | 1.05 MiB/s, done. Total 7 (delta 0), reused 0 (delta 0) remote: remote: To create a merge request for develop, visit: remote: http://localhost/root/spring-boot-helloworld-deployment/-/merge_requests/new?merge_request%5Bsource_branch%5D=develop remote: To http://git.ik8s.com/root/spring-boot-helloworld-deployment.git a832bd7..5f11ced master -> master * [new branch] develop -> develop ``` ![image-20210514192028861](gitops.assets/image-20210514192028861.png) ### Jenkins 自动部署操作 在 jenkins 上新建一个项目，使其完成自动部署操作： ![image-20210514192321396](gitops.assets/image-20210514192321396.png) 配置 pipeline 脚本： > 注意更换 git 仓库地址 ```groovy pipeline { agent { kubernetes { inheritFrom 'kubectl' } } stages { stage('Source') { steps { git branch: 'develop', url: 'http://gitlab.gitops.svc.cluster.local/root/spring-boot-helloworld-deployment.git' } } stage('deployment') { steps { container('kubectl') { withKubeConfig([credentialsId:'k8s-cluster-admin-kubeconfig']) { sh 'kubectl apply -f kubernetes/' } } } } } } ``` ![image-20210514193650053](gitops.assets/image-20210514193650053.png) 上面的 pipeline 里面调用了一个 credentialsId，它的类型是一个文件，这文件指明了我们的集群地址，以及集群的读写权限，可以认证到 k8s 集群里面去部署相关资源，需要去 jenkins 凭据里面创建 credentialsId 并上传认证文件： 导出 k8s 认证文件，使用 `kubectl config view --raw` 命令即可看到相关信息，可直接保存为文件： ```bash root@k8s01 ~# kubectl config view --raw > cluster-admin.kubeconfig ``` 该文件和家目录下的 `.kube/config` 一致，也可以使用家目录下的 `.kube/config` 文件。 cluster-admin.kubeconfig 文件里面的地址是集群外的地址，也可以使用集群内的地址： ```bash # 把 server: https:// 后面的地址换为 svc 地址 server: https://192.168.0.101:6443 # 查看 k8s 集群的 svc 入口 root@k8s01 ~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.68.0.1 443/TCP 24d # 修改地址如下，注意端口，集群内访问的是 443 端口 server: https://kubernetes.default.svc.cluster.local:443 ``` 把 `cluster-admin.kubeconfig` 或 `.kube/config` 文件上传到 jenkins 里面： 我这里是 通过 sz 命令 把 `cluster-admin.kubeconfig` 文件传到 windows ，在使用浏览器上传到 jenkins： ![image-20210514195341478](gitops.assets/image-20210514195341478.png) 在 jenkins 添加 凭据，并上传该文件，ID 一定要和 脚本里写的一致： ![image-20210514195732803](gitops.assets/image-20210514195732803.png) ![image-20210514195816589](gitops.assets/image-20210514195816589.png) 接下来添加一个 Pod Template ，里面的容器可以使用 `kubectl` 命令： ![image-20210514200440059](gitops.assets/image-20210514200440059.png) 返回 jenkins spring-boot-helloworld-deployment 项目，手动触发构建，点击立即构建： ![image-20210514201902346](gitops.assets/image-20210514201902346.png) 可以看到构建成功了，去 k8s 集群看看是否部署了相关的 pod，service 等资源： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# kubectl get all -n helloworld NAME READY STATUS RESTARTS AGE pod/helloworld-7df8f7cd85-65xfp 1/1 Running 0 11m pod/helloworld-7df8f7cd85-82hg9 1/1 Running 0 11m pod/helloworld-7df8f7cd85-mhjgg 1/1 Running 0 11m pod/helloworld-7df8f7cd85-zdxq5 1/1 Running 0 11m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/helloworld ClusterIP 10.68.8.95 80/TCP 11m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/helloworld 4/4 4 4 11m NAME DESIRED CURRENT READY AGE replicaset.apps/helloworld-7df8f7cd85 4 4 4 11m ``` 解析一下域名，应该就可以使用浏览器访问了： ![image-20210514202123816](gitops.assets/image-20210514202123816.png) ![image-20210514202229782](gitops.assets/image-20210514202229782.png) ![image-20210514202255385](gitops.assets/image-20210514202255385.png) 如果出错了下面的错误，可能是缺了 kubernetes CLI 插件： ![image-20210514201848491](gitops.assets/image-20210514201848491.png) ### 配置 Jenkins 经由 Gitlab 触发 配置 jenkins spring-boot-helloworld 项目： ![image-20210517111419613](gitops.assets/image-20210517111419613.png) 上面页面下拉，点击 **高级** 配置，找到 **Secret token** 点击 **Generate** 创建一个 Secret token，注意另外保存这串密文： ![image-20210517111603068](gitops.assets/image-20210517111603068.png) > GitLab webhook URL: http://jenkins.ik8s.com/project/spring-boot-helloworld-deployment > > Secret token: 61bb7f1eabef1532e6722ec658ce038b ### 配置 Gitlab Webhooks 去 Gitlab 页面找到 spring-boot-helloworld 项目，填写 jenkins 地址和 Secret token： jenkins 地址就是 配置 构建触发器 时 显示的那个地址，这里可以改为集群内的 svc 地址，注意 svc 的端口。 > GitLab webhook URL: http://jenkins.gitops.svc.cluster.local/project/spring-boot-helloworld-deployment > > Secret token: 61bb7f1eabef1532e6722ec658ce038b ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# kubectl get svc -n gitops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... jenkins ClusterIP 10.68.231.40 8080/TCP,50000/TCP 8d ... ``` ![image-20210517113641248](gitops.assets/image-20210517113641248.png) ![image-20210517113253739](gitops.assets/image-20210517113253739.png) 页面刷新后下面会出现一个 Project Hooks (1)，手动点击测试，会看到 jenkins spring-boot-helloworld-deployment 自动构建： ![image-20210517113809466](gitops.assets/image-20210517113809466.png) ![image-20210517114038444](gitops.assets/image-20210517114038444.png) 说明 gitlab 上传代码后，触发 jenkins 自动构建配置成功。 ### 使用Jenkinsfile进行构建 把 jenkins 脚本加入到代码一起打包上传至代码仓库： 在 spring-boot-helloworld-deployment 代码根目录下添加一个 Jenkinsfile 文件： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# ls Jenkinsfile kubernetes README.md ``` Jenkinsfile 内容如下： ```groovy pipeline { agent { kubernetes { inheritFrom 'kubectl' } } stages { stage('Source') { steps { git branch: 'develop', url: 'http://gitlab.gitops.svc.cluster.local/root/spring-boot-helloworld-deployment.git' } } stage('deployment') { steps { container('kubectl') { withKubeConfig([credentialsId:'k8s-cluster-admin-kubeconfig']) { sh 'kubectl apply -f kubernetes/' } } } } } } ``` 重新配置 jenkins 上的 spring-boot-helloworld-deployment 的配置： ![image-20210517110600359](gitops.assets/image-20210517110600359.png) ![image-20210517145415302](gitops.assets/image-20210517145415302.png) 至此 cicd 就基本完成。 ### 模拟完整的 CICD 流水线 开发人员修改，上传代码： ```bash # 修改代码 root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# ls deploy Dockerfile Jenkinsfile LICENSE pom.xml README.md src root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep SNAPSHOT pom.xml 0.9.1-SNAPSHOT root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# sed -i 's/0.9.1-SNAPSHOT/0.9.2-SNAPSHOT/' pom.xml root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep SNAPSHOT pom.xml 0.9.2-SNAPSHOT root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep 0.9.1 src/main/java/com/neo/controller/HelloWorldController.java return "version 0.9.1"; root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# sed -i 's/0.9.1/0.9.2/' src/main/java/com/neo/controller/HelloWorldController.java root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep 0.9. src/main/java/com/neo/controller/HelloWorldController.java return "version 0.9.2"; root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep v0.9. Jenkinsfile appVersion = "v0.9.1" root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# sed -i 's/0.9.1/0.9.2/' Jenkinsfile root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# grep v0.9. Jenkinsfile appVersion = "v0.9.2" # 上传代码 root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git add . root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git commit -m "update to v0.9.2" [develop 34b993b] update to v0.9.2 2 files changed, 2 insertions(+), 2 deletions(-) root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld# git push origin develop Username for 'http://git.ik8s.com': root Password for 'http://root@git.ik8s.com': Counting objects: 10, done. Delta compression using up to 4 threads. Compressing objects: 100% (8/8), done. Writing objects: 100% (10/10), 739 bytes | 739.00 KiB/s, done. Total 10 (delta 3), reused 0 (delta 0) remote: remote: To create a merge request for develop, visit: remote: http://localhost/root/spring-boot-helloworld/-/merge_requests/new?merge_request%5Bsource_branch%5D=develo remote: To http://git.ik8s.com/root/spring-boot-helloworld.git 4a49447..34b993b develop -> develop ``` 查看 git jenkins harbor spring-boot-helloworld 项目是否有对应的数据： ![image-20210517151822658](gitops.assets/image-20210517151822658.png)![image-20210517151629520](gitops.assets/image-20210517151629520.png) ![image-20210517151705385](gitops.assets/image-20210517151705385.png) 运维人员修改 容器部署清单： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# vim kubernetes/ 1-ns.yaml 2-svc.yaml 3-ingress.yaml 4-deployment.yaml root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# grep v0.9. kubernetes/4-deployment.yaml image: 'registry.ik8s.com/spring-boot-helloworld/spring-boot-helloworld:v0.9.1' root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# sed -i 's/0.9.1/0.9.2/' kubernetes/4-deployment.yaml root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# grep v0.9. kubernetes/4-deployment.yaml image: 'registry.ik8s.com/spring-boot-helloworld/spring-boot-helloworld:v0.9.2' ``` 运维上传 清单到 git 仓库： ```bash root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# git add . root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# git commit -m "update to v0.9.2" [develop bf24a8b] update to v0.9.2 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 Jenkinsfile root@k8s01 ~/k8s/cicd/gitops/spring-boot-helloworld-deployment# git push origin develop Username for 'http://git.ik8s.com': root Password for 'http://root@git.ik8s.com': Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (5/5), done. Writing objects: 100% (5/5), 684 bytes | 684.00 KiB/s, done. Total 5 (delta 2), reused 0 (delta 0) remote: remote: To create a merge request for develop, visit: remote: http://localhost/root/spring-boot-helloworld-deployment/-/merge_requests/new?merge_request%5Bsource_branch%5D=develop remote: To http://git.ik8s.com/root/spring-boot-helloworld-deployment.git 5f11ced..bf24a8b develop -> develop ``` 查看 git jenkins spring-boot-helloworld-deployment 项目： ![image-20210517152437044](gitops.assets/image-20210517152437044.png) ![image-20210517152503176](gitops.assets/image-20210517152503176.png) 至此 CICD 流程就完成。还可以更完善。