TLS 与引导启动原理
在 K8s 中,为了简化证书的生成,kubelet 可以主动请求 api-server 来为自己颁发证书。
官网讲的比我更详细,建议跟着官网走,这里仅供参考。
配置 kube-apiserver
配置令牌认证文件
需要生成一个令牌,令牌可以为任意数据,但推荐使用下面的命令生成:
head -c 16 /dev/urandom | od -An -t x | tr -d ' 'bash
上面的命令会生成类似于 cdbe50bd844b6952cd197899bac486d9
的令牌,之后组装成令牌文件:
cdbe50bd844b6952cd197899bac486d9,kubelet-bootstrap,10001,"system:bootstrappers"text
具体格式为:令牌,用户名,用户UID,用户组(多个使用双引号括起来)
之后修改 kube-apiserver 参数,添加 --token-auth-file=FILENAME
标志,然后重启:
systemctl daemon-reload && systemctl restart kube-apiserver && systemctl status kube-apiserverbash
给授权用户组
配置好 kubectl 命令的认证文件后(~/.kube/config
,可以直接复制 /etc/kubernetes/admin.conf
),应用下面的 yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: create-csrs-for-bootstrapping
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:node-bootstrapper
apiGroup: rbac.authorization.k8s.ioyaml
即使没有任何节点,这个文件也是可以直接应用的。
配置 kube-controller-manager
由于签发证书是 kube-controller-manager
实现的,所以必须保证启动时配置了合法的 --cluster-signing-cert-file
和 --cluster-signing-key-file
。
应用下面的 yaml,第一个是允许新节点自动生成新的证书:
# 批复 "system:bootstrappers" 组的所有 CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: auto-approve-csrs-for-group
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiGroup: rbac.authorization.k8s.ioyaml
这个是允许节点对证书进行续期:
# 批复 "system:nodes" 组的 CSR 续约请求
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.ioyaml
kubelet 配置
在执行前需要节点中准备签发 apiserver 证书的 CA 证书:
ssh k8s-node-1 "mkdir -p /etc/kubernetes/pki; mkdir -p /etc/kubernetes/manifests"
scp /etc/kubernetes/pki/ca.crt root@k8s-node-1:/etc/kubernetes/pki/ca.crtbash
创建一个 kubelet context
:
kube=/etc/kubernetes/bootstrap-kubeconfig
KUBECONFIG=$kube kubectl config set-context default-auth@kubernetes --cluster=kubernetes --user=default-auth
# 注意 server 配负载均衡地址
KUBECONFIG=$kube kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.crt \
--embed-certs=true \
--server=https://192.168.1.34:6443
KUBECONFIG=$kube kubectl config set-credentials default-auth \
--cluster=kubernetes \
--token=cdbe50bd844b6952cd197899bac486d9
KUBECONFIG=$kube kubectl config use-context default-auth@kubernetesbash
在启动 kubelet 前,要准备好 containerd 的配置,因为默认这个东西是禁用 cri 的:
containerd config default > /etc/containerd/config.tomlbash
然后修改 sandbox_image
为 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:<your_version>
。
创建 kubelet 配置文件(/etc/kubernetes/kubelet-conf.yaml
):
# clusterDNS 为 service 网络的第 10 个 ip 值,如 10.96.0.10
cat << EOF > /etc/kubernetes/kubelet-conf.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
serializeImagePulls: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.0
clusterDomain: cluster.local
staticPodPath: /etc/kubernetes/manifests
EOFyaml
所有的配置文档可以在这里找到:KubeletConfiguration。
创建 kubelet 服务并启动:
cat << EOF > /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=
After=containerd.service
Requires=containerd.server
[Service]
ExecStart=/usr/local/bin/kubelet \\
--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubeconfig \\
--kubeconfig=/etc/kubernetes/kubeconfig \\
--config=/etc/kubernetes/kubelet-conf.yaml
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now kubelet
systemctl status kubeletbash
--kubeconfig
所指向的文件并不需要存在,只需要它是一个合法的路径即可,这个文件会被自动生成。
所有节点执行完成后,就可以看到节点状态了:
[root@k8s-master-1 kubernetes]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master-1 NotReady <none> 92s v1.29.6 k8s-master-2 NotReady <none> 3m18s v1.29.6 k8s-master-3 NotReady <none> 2m28s v1.29.6 k8s-node-1 NotReady <none> 46m v1.29.6 k8s-node-2 NotReady <none> 7m8s v1.29.6 k8s-node-3 NotReady <none> 4m31s v1.29.6bash
kube-proxy 启动
生成 kube-proxy.conf
在主节点执行:
# 创建 kube-proxy 的服务账号
kubectl -n kube-system create serviceaccount kube-proxy
# 创建角色绑定
kubectl create clusterrolebinding system:kube-proxy \
--clusterrole system:node-proxier \
--serviceaccount kube-system:kube-proxy
# 创建长效 API 令牌
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: kube-proxy-secret
namespace: kube-system
annotations:
kubernetes.io/service-account.name: kube-proxy
type: kubernetes.io/service-account-token
EOF
# 导出为变量,方便后续使用
JWT_TOKEN=$(kubectl get -n kube-system secret/kube-proxy-secret --output=jsonpath='{.data.token}' | base64 --decode)
KUBECONFIG=/etc/kubernetes/kube-proxy.conf
# 创建配置文件
kubectl config set-context kube-proxy@kubernetes \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=$KUBECONFIG
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.crt \
--embed-certs=true \
--server=https://192.168.1.34:6443 \
--kubeconfig=$KUBECONFIG
kubectl config set-credentials kube-proxy \
--token=$JWT_TOKEN \
--kubeconfig=$KUBECONFIG
kubectl config use-context kube-proxy@kubernetes --kubeconfig=$KUBECONFIGbash
创建配置文件(kube-proxy 配置 (v1alpha1) 资源类型):
cat << EOF > /etc/kubernetes/kube-proxy.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
bindAddress: 0.0.0.0
oomScoreAdj: -999
clientConnection:
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /etc/kubernetes/kube-proxy.conf
clusterCIDR: 196.16.0.0/16 # Pod cidr
configSyncPeriod: 15m0s
conntrack:
min: 131072
tcpEstablishedTimeout: 30m0s
tcpCloseWaitTimeout: 2m0s
ipvs:
syncPeriod: 30s
scheduler: rr
minSyncPeriod: 5s
EOFyaml
创建完后,将 /etc/kubernetes/kube-proxy.conf
和 /etc/kubernetes/kube-proxy.yaml
发送给所有节点。
创建服务
创建服务:
cat << EOF > /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/kube-proxy/
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy.yaml \
--master=https://192.168.1.34:6443 \
--v=2
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now kube-proxy
systemctl status kube-proxybash
如果现在使用 kubectl describe
来查看节点,可以发现:
Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Tue, 23 Jul 2024 00:01:28 +0800 Fri, 19 Jul 2024 23:34:16 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 23 Jul 2024 00:01:28 +0800 Fri, 19 Jul 2024 23:34:16 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 23 Jul 2024 00:01:28 +0800 Fri, 19 Jul 2024 23:34:16 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Tue, 23 Jul 2024 00:01:28 +0800 Fri, 19 Jul 2024 23:34:16 +0800 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNolog
至此集群已经搭建完毕,下一步则是安装网络插件。