这篇文章主要介绍“K8s集群部署高可用架构”，在日常操作中，相信很多人在K8s集群部署高可用架构问题上存在疑惑，小编查阅了各式资料，整理出简单好用的操作方法，希望对大家解答”K8s集群部署高可用架构”的疑惑有所帮助！接下来，请跟着小编一起来学习吧！
创新互联建站服务项目包括武汉网站建设、武汉网站制作、武汉网页制作以及武汉网络营销策划等。多年来，我们专注于互联网行业，利用自身积累的技术优势、行业经验、深度合作伙伴关系等，向广大中小型企业、政府机构等提供互联网行业的解决方案，武汉网站推广取得了明显的社会效益与经济效益。目前，我们服务的客户以成都为中心已经辐射到武汉省份的部分城市，未来相信会继续扩大服务区域并继续获得客户的支持与信任！
环境
系统 角色 IP centos7.4 master-1 10.10.25.149 centos7.4 master-2 10.10.25.112 centos7.4 node-1 10.10.25.150 centos7.4 node-2 10.10.25.151 centos7.4 lb-1 backup 10.10.25.111 centos7.4 lb-2 master 10.10.25.110 centos7.4 VIP 10.10.25.113
部署master02 节点
重庆分公司
重庆分公司
拷贝master01上面的 /opt/kubernetes/目录
scp -r /opt/kubernetes/ root@10.10.25.112:/opt
拷贝master01上的相关服务
scp /usr/lib/systemd/system/{kube-apiserver,kube-scheduler,kube-controller-manager}.service root@10.10.25.112:/usr/lib/systemd/system
vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
--bind-address=10.10.25.112 \
--insecure-bind-address=127.0.0.1 \
--authorization-mode=Node,RBAC \
--runtime-config=rbac.authorization.k8s.io/v1 \
--kubelet-https=true \
--anonymous-auth=false \
--basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
--service-cluster-ip-range=10.1.0.0/16 \
--service-node-port-range=20000-40000 \
--tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \
--etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \
--etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \
--etcd-servers=https://10.10.25.149:2379,https://10.10.25.150:2379,https://10.10.25.151:2379 \
--enable-swagger-ui=true \
--allow-privileged=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/log/api-audit.log \
--event-ttl=1h \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
启动apiserver
systemctl start kube-apiserver
# ps -aux | grep kube
systemctl start kube-scheduler kube-controller-manager
加入系统path
vim /root/.bash_profile
添加
PATH=$PATH:$HOME/bin:/opt/kubernetes/bin
source .bash_profile查看组件状态
# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
此时已经说明可以连接到ETCD集群查看node状态
# /opt/kubernetes/bin/kubectl get node NAME STATUS ROLES AGE VERSION 10.10.25.150 NotReady14d v1.10.3 10.10.25.151 NotReady 14d v1.10.3 说明master02 还无法与node通信
配置单节点LB负载均衡
注:做高可用集群时间上需要同步
lb02节点配置
配置nginx yum源,使用4层代理做
vim /etc/yum.repos.d/nginx.repo [nginx] name=nginx repo baseurl=https://nginx.org/packages/centos/7/$basearch/ gpgcheck=0 enabled=1 yum install -y nginx
修改Nginx配置文件
vim /etc/nginx/nginx.conf
stream {
log_format main "remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 10.10.25.149:6443;
server 10.10.25.112:6443;
}
server {
listen 10.10.25.110:6443;
proxy_pass k8s-apiserver;
}
}修改node节点
cd /opt/kubernetes/cfg/ vim bootstrap.kubeconfig 修改 server: https://10.10.25.149:6443 为 server: https://10.10.25.110:6443 vim kubelet.kubeconfig 修改 server: https://10.10.25.149:6443 为 server: https://10.10.25.110:6443 vim kube-proxy.kubeconfig 修改 server: https://10.10.25.149:6443 为 server: https://10.10.25.110:6443 systemctl restart kubelet systemctl restart kube-proxy
此时启动以后会发现master01 master02 都无法与node节点通讯,查看node日志发现,提示证书错误,大致意思是kube-proxy证书是master01节点的而不是LB节点的.所以接下来我们需要重新生成Kube-proxy证书
master01重新生成api-server证书
编辑证书json文件
[root@master ssl]# cat kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.10.25.149",
"10.10.25.112",
"10.10.25.110",
"10.10.25.111",
"10.10.25.113",
"10.1.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
说明:json文件中的IP包括master01 master02节点IP,所有LB节点IP和VIP 的地址,因为我们最终需要实现 Nginx + Keepalive 0单节点的负载均衡架构
生成证书
cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
拷贝到相应节点
cp kubernetes*.pem /opt/kubernetes/ssl/
scp kubernetes*.pem 10.10.25.112:/opt/kubernetes/ssl/
scp kubernetes*.pem 10.10.25.150:/opt/kubernetes/ssl/
scp kubernetes*.pem 10.10.25.151:/opt/kubernetes/ssl/重启master节点的服务
systemctl start kube-scheduler kube-controller-manager kube-apiserver
重启node节点服务
systemctl restart kube-proxy kubelet
验证
# kubectl get node NAME STATUS ROLES AGE VERSION 10.10.25.150 Ready15d v1.10.3 10.10.25.151 Ready 15d v1.10.3 说明已经实现了单节点负载均衡. 这里有个地方需要注意,在以上配置都完成并且没有错误的情况下,有可能出现获取到的node状态是notready,有可能出现此问题的有原因是在日志里面发现node无法注册,此时我们需要手动注册,在master01上执行一下命令即可 kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve
lb01节点配置
同样安装nginx,过程不赘述，nginx配置也相同。需要改变的只是绑定的IP
vim /etc/nginx/nginx.conf
stream {
log_format main "remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 10.10.25.149:6443;
server 10.10.25.112:6443;
}
server {
listen 10.10.25.111:6443;
proxy_pass k8s-apiserver;
}
}使用Keepalive实现LB节点的高可用
安装keepalive两个节点都需要
yum install keepalived -y
设置lb02为keepalived为master节点
修改lb02keepalived配置文件
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh" #脚本检查ngixn状态
}
vrrp_instance VI_1 {
state MASTER
interface ens192
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.10.25.113/24
}
track_script {
check_nginx
}
}编写nginx状态检测脚本
cat /etc/keepalived/check_nginx.sh #!/bin/sh count=$(ps -ef | grep nginx | egrep -cv "grep|$$") #获取nginx进程数 if [ "$count" -eq 0 ];then systemctl stop keepalived fi 授予脚本执行权限 chmod +x /etc/keepalived/check_nginx.sh
启动keepalived
systemctl start keepalived
查看是VIP是否生效
