提供了编程的基础技术教程

zl程序教程

您现在的位置是：首页 >  工具

当前栏目

eks 使用案例 部署jenkins

Jenkins案例部署 使用
2023-09-14 09:08:38 时间

https://aws.amazon.com/cn/blogs/storage/deploying-jenkins-on-amazon-eks-with-amazon-efs/

这个链接挺好的，包含了，使用aws cli
创建 eks
创建安全组
创建EFS存储
挂在存储
使用helm安装jenkins等， helm的网址https://artifacthub.io/packages/helm/bitnami/jenkins

前期准备工作：

1）有一个aws账户
不赘述

2）在账户中购买一个迷你主机，并在主机上安装aws cli，并且配置aws configure

安装aws cli

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

**配置aws cli凭证, 如何获取凭证，这里不多赘述，主要是在iam用户中设置

$ aws configure
AWS Access Key ID [None]: <AKIAIOSFODNN7EXAMPLE>
AWS Secret Access Key [None]: <wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY>
Default region name [None]: <region-code>
Default output format [None]: <json>

3） 安装aws-iam-authenticator

curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.18.9/2020-11-02/bin/linux/amd64/aws-iam-authenticator
chmod +x ./aws-iam-authenticator
mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$PATH:$HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc

4）安装EKS 的命令行管理工具 kubectl

curl -o kubectl https://s3.us-west-2.amazonaws.com/amazon-eks/1.22.6/2022-03-09/bin/linux/amd64/kubectl
chmod +x ./kubectl
mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
kubectl version --short --client

5）安装eksctl (一个可以通过命令简单创建和管理 Amazon EKS的工具)

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

6）安装Helm

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

正式开始

1）创建Amazon EKS cluster

ssh-keygen -t rsa
#回车三连
#创建私钥
cat << "EOF" > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: faberbeta
  region: ap-east-1
  version: "1.22"

managedNodeGroups:
  - name: nodegroup01
    maxPodsPerNode: 200
    instanceType: m5.large
    minSize: 3
    maxSize: 6
    desiredCapacity: 3
    volumeType: gp3
    volumeSize: 80
    ssh:
      allow: true
      publicKeyPath: ~/.ssh/id_rsa.pub
    labels: {role: worker}
    tags:
      nodegroup-role: worker
    iam:
      withAddonPolicies:
        imageBuilder: true
        autoScaler: true
        externalDNS: true
        certManager: true
        appMesh: true
        appMeshPreview: true
        ebs: true
        fsx: true
        efs: true
        albIngress: true
        xRay: true
        cloudWatch: true
EOF

eksctl create cluster -f cluster.yaml

该命令会在ap-east-1创建一个新的EKS cluster（faberbeta）并且创建一个EKS-managed nodegroup(nodegroup001)
nodegroup会创建三个m5.large实例的ec2 , 硬盘80G
该provisioning需要大概10-15分钟

测试kubectl 配置是否正确

kubectl get svc

输出

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   6m20s

2）创建一个Amazon EFS 文件系统

1）获取你新建eks 集群所生成的 VPC ID （IP段是192.168.0.0/16）
aws ec2 describe-vpcs
或者在后台查看

  1. 为你的Amazon EFS mount target设置一个安全组
aws ec2 create-security-group \
--region ap-east-1 \
--group-name efs-mount-sg \
--description "Amazon EFS for EKS, SG for mount target" \
--vpc-id vpc-0828d8cd5a6f58db3

3）给刚才新建的安全组设置规则
给inbound 流量开放 NFS port (2049)

aws ec2 authorize-security-group-ingress \
--group-id sg-0ee2e7fd637999755 \
--region ap-east-1 \
--protocol tcp \
--port 2049 \
--cidr 192.168.0.0/16
  1. 创建Amazon EFS 文件系统
aws efs create-file-system \
--creation-token creation-token \
--performance-mode generalPurpose \
--throughput-mode bursting \
--region ap-east-1 \
--tags Key=Name,Value=MyEFSFileSystem \
--encrypted

5）获取你新建的EKS下的实例的 VPC subnet IDs

aws ec2 describe-instances \
 --filters Name=vpc-id,Values=vpc-009e248a7d4834f00 \
 --query 'Reservations[*].Instances[].SubnetId'

因为新建了两个EC2实例，并且这两个实例分属独立的k8s虚拟网络subnet中
会得到两个 subnet ID
输出结果

[
    "subnet-0f8061a97f9e96d11",
    "subnet-02266afc033a04a26"
]
  1. 建立两个 Amazon EFS mount targets
    第一个
aws efs create-mount-target \
--file-system-id fs-f2ae293f \
--subnet-id subnet-0f8061a97f9e96d11 \
--security-group sg-0ee2e7fd637999755 \
--region ap-east-1

第二个

aws efs create-mount-target \
--file-system-id fs-f2ae293f \
--subnet-id subnet-02266afc033a04a26 \
--security-group sg-0ee2e7fd637999755 \
--region ap-east-1

*注意在刚才获取的两个可用区都创建Amazon EFS mount targets
7) 现在创建Amazon EFS access point
Now that you have your file system, let’s create an Amazon EFS Access Point. Amazon EFS access points are application-specific entry points into an EFS file system that make it easier to manage application access to shared datasets or, in our case, configuration. Regardless of how a container is built, access points can enforce a user identity, including the user’s POSIX groups, for all file system requests that are made through them. For our purposes, let’s create a Jenkins-specific EFS access point and choose to enforce user ID and a group ID of 1000 using the following command:

aws efs create-access-point --file-system-id fs-f2ae293f \
--posix-user Uid=1000,Gid=1000 \
--root-directory "Path=/jenkins,CreationInfo={OwnerUid=1000,OwnerGid=1000,Permissions=777}"

记录 access point ID (that is, fsap-0123456abc987634a) 给之后使用

给EKS集群部署 Amazon EFS CSI driver

1） 部署Amazon EFS CSI driver,

sudo yum install git -y
kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

输出

serviceaccount/efs-csi-controller-sa created
clusterrole.rbac.authorization.k8s.io/efs-csi-external-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/efs-csi-provisioner-binding created
deployment.apps/efs-csi-controller created
daemonset.apps/efs-csi-node created
csidriver.storage.k8s.io/efs.csi.aws.com configured
  1. 创建efs-sc storage class YAML 文件
    随着Amazon EFS CSI driver 的安装，您可以创建一个存储类，使您能够配置持久卷，以允许pod使用amazon efs文件系统。
    复制下面的配置，并且保存为storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com

3）创建efs-pv persistent volume YAML 文件
让我们继续为我们的Jenkins应用程序创建一个persistent volume和一个persistent volume claim。首先，复制以下配置并将其保存到名为persistentvolume.yaml 确保将volumeHandle参数修改为file system ID and access point ID:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: identifier for our file system::identifier for our 
access point (i.e. fs-123b45fa::fsap-12345678910ab12cd34)
  1. 创建efs-claim persistent volume claim YAML 文件
    复制下面的，并且命名为persistentvolumeclaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

注意：因为amazon EFS是一个弹性文件系统，所以它不强制执行任何文件系统容量限制。创建文件系统时不使用持久卷和持久卷声明中的实际存储容量值。但是，由于存储容量是Kubernetes中的必填字段，因此必须指定有效值，例如本例中的5Gi。此值不限制Amazon EFS文件系统的大小。

  1. 部署efs-sc storage class, efs-pv persistent volume, and efs-claim persistent volume claim
kubectl apply -f \
storageclass.yaml,persistentvolume.yaml,persistentvolumeclaim.yaml
  1. 确保Kubernetes 的资源已经创建

部署 Jenkins 到 Amazon EKS 中

在本章节中将使用 HELM（Kubernetes的包管理器，帮助您在Kubernetes集群上安装和管理应用程序。）部署jenkins 到amazon EKS中
1） 添加Helm stable chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami
2） 在你的EKS cluster上安装jenkins

helm install jenkins bitnami/jenkins --set \
rbac.create=true,master.servicePort=80,master.serviceType=NodePort,persistence.existingClaim=efs-claim
  1. 获取ingress loadbalancer name
    部署 ingress nginx
    根据官网部署
    https://kubernetes.github.io/ingress-nginx/deploy/#aws
1. Download the the deploy-tls-termination.yaml template:

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/aws/deploy-tls-termination.yaml

2. Edit the file and change the VPC CIDR in use for the Kubernetes cluster:

proxy-real-ip-cidr: XXX.XXX.XXX/XX


3. Change the AWS Certificate Manager (ACM) ID as well:

arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX

ingress之NLB配置固定IP EIP
  1. aws-load-balancer-connection-idle-timeout: '60' 修改为350秒
  2. Deploy the manifest:
kubectl apply -f deploy-tls-termination.yaml

ingres 配置 示例，请根据实际server名和端口和白名单等修改

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-jenkins
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "1。2.3.4,1.1.1.1" #定义白名单访问
    nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'false'
spec:
  rules:
    - host: jenkins.xxx.com
      http:
        paths:
        - path: /
          backend:
            serviceName: jenkins
            servicePort: 80

相关文章