云计算 K8S使用dashboard管理集群
Heapster介绍今年3月份在公司的内部k8s培训会上，开发同事表示使用dashboard的可以满足日常开发需求，例如查看pod的日志，执行exec指令，查看pod的运行状态等，但对basic认证的权限控制表示担忧。
之前介绍过在1.5.2版本上部署dashboard服务，在1.9.1版本离线部署中，也介绍过dashboard服务的RBAC配置和使用技巧。因此本文将在前文基础上完善Heapster的整合与利用token对用户权限进行控制。
dashboard的特点主要如下：
1、能够直观的看到rc、deployment、pod、services等k8s组件的运行情况和日志信息。
2、结合heapster和influxdb后，dashboard的监控图表上可以看到pod的cpu和内存消耗情况。
1、Heapster是容器集群监控和性能分析工具，支持Kubernetes和CoreOS。
2、K8S集群的HPA功能的实现就依赖于这些metric数据，HPA将Heapster作为Resource Metrics API，向其获取metric。
3、Kubernetes有个cAdvisor监控（在1.9版本里面，cAdvisor已经和kubelet整合在一起）。
在每个kubernetes
Node上都会运行cAdvisor，它会收集本机以及容器的监控数据(cpu,memory,filesystem,network,uptime)。Heapster是一个收集者，Heapster可以收集Node节点上的cAdvisor数据，将每个Node上的cAdvisor的数据进行汇总，还可以按照kubernetes的资源类型来集合资源，比如Pod、Namespace，可以分别获取它们的CPU、内存、网络和磁盘的metric。默认的metric数据聚合时间间隔是1分钟。还可以把数据导入到第三方工具(如InfluxDB)。
2、Influxdb数据库的相关知识介绍，可参考文档：https://www.jianshu.com/p/d2935e99006e
2、如果对Heapster收集到的metric数据没有持久化的需求，可以不配置Influxdb数据库
3、本文Influxdb数据库的存储采用emptydir的方式实现，实际使用过程中，可以选择吧Influxdb数据库部署在k8s集群外部，或者使用其他存储方案。
4、如果有需要的话，还可以集成一个grafana做web展示。Grafana配置可参考文档：https://blog.51cto.com/ylw6006/2084403
需要科学上网方式获取到dashboard相关的镜像文件，仓库可纳入本地仓库统一管理
#cat/etc/systemd/system/docker.service.d/http-proxy.conf[Service] Environment="HTTP_PROXY=http://192.168.115.2:1080"#systemctldaemon-reload#systemctlrestartdocker#dockerpullk8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3#dockerpullk8s.gcr.io/heapster-influxdb-amd64:v1.3.3#dockerpullk8s.gcr.io/heapster-amd64:v1.4.2二、准备配置文件
1、k8s-dashborad-sa.yaml文件，secrct和serviceaccount配置
#catk8s-dashborad-sa.yaml#-------------------DashboardSecret-------------------#apiVersion:v1kind:Secretmetadata: labels: k8s-app:kubernetes-dashboard name:kubernetes-dashboard-certs namespace:kube-systemtype:Opaque---#-------------------DashboardServiceAccount-------------------#apiVersion:v1kind:ServiceAccountmetadata: labels: k8s-app:kubernetes-dashboard name:kubernetes-dashboard namespace:kube-system
2、k8s-dashborad-rbac.yaml文件，配置 Role和Role Binding
#catk8s-dashborad-rbac.yaml#-------------------DashboardRole&RoleBinding-------------------#kind:RoleapiVersion:rbac.authorization.k8s.io/v1metadata: name:kubernetes-dashboard-minimal namespace:kube-systemrules: #AllowDashboardtocreate'kubernetes-dashboard-key-holder'secret.-apiGroups:[""] resources:["secrets"] verbs:["create"]#AllowDashboardtocreate'kubernetes-dashboard-settings'configmap.-apiGroups:[""] resources:["configmaps"] verbs:["create"]#AllowDashboardtoget,updateanddeleteDashboardexclusivesecrets.-apiGroups:[""] resources:["secrets"] resourceNames:["kubernetes-dashboard-key-holder","kubernetes-dashboard-certs"] verbs:["get","update","delete"]#AllowDashboardtogetandupdate'kubernetes-dashboard-settings'configmap.-apiGroups:[""] resources:["configmaps"] resourceNames:["kubernetes-dashboard-settings"] verbs:["get","update"]#AllowDashboardtogetmetricsfromheapster.-apiGroups:[""] resources:["services"] resourceNames:["heapster"] verbs:["proxy"] -apiGroups:[""] resources:["services/proxy"] resourceNames:["heapster","http:heapster:","https:heapster:"] verbs:["get"] ---apiVersion:rbac.authorization.k8s.io/v1kind:RoleBindingmetadata: name:kubernetes-dashboard-minimal namespace:kube-systemroleRef: apiGroup:rbac.authorization.k8s.io kind:Role name:kubernetes-dashboard-minimalsubjects:-kind:ServiceAccount name:kubernetes-dashboard namespace:kube-system
3、k8s-dashborad-deployment.yaml配置文件，定义创建pod的模板和副本数
重庆分公司
重庆分公司
#catk8s-dashborad-deployment.yaml#-------------------DashboardDeployment-------------------#kind:DeploymentapiVersion:apps/v1beta2metadata:
labels:
k8s-app:kubernetes-dashboard
name:kubernetes-dashboard
namespace:kube-systemspec:
replicas:1
revisionHistoryLimit:10
selector:
matchLabels:
k8s-app:kubernetes-dashboard
template:
metadata:
labels:
k8s-app:kubernetes-dashboard
spec:
containers:
-name:kubernetes-dashboard
image:k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
ports:
-containerPort:8443
protocol:TCP
args:
---auto-generate-certificates#UncommentthefollowinglinetomanuallyspecifyKubernetesAPIserverHost
#Ifnotspecified,DashboardwillattempttoautodiscovertheAPIserverandconnect
#toit.Uncommentonlyifthedefaultdoesnotwork.
#---apiserver-host=http://my-address:port
volumeMounts:
-name:kubernetes-dashboard-certs
mountPath:/certs#Createon-diskvolumetostoreexeclogs
-mountPath:/tmp
name:tmp-volume
livenessProbe:
httpGet:
scheme:HTTPS
path:/
port:8443
initialDelaySeconds:30
timeoutSeconds:30
volumes:
-name:kubernetes-dashboard-certs
secret:
secretName:kubernetes-dashboard-certs
-name:tmp-volume
emptyDir:{}
serviceAccountName:kubernetes-dashboard#CommentthefollowingtolerationsifDashboardmustnotbedeployedonmaster
tolerations:
-key:node-role.kubernetes.io/master
effect:NoSchedule4、 k8s-dashborad-service.yaml配置文件，定义service
#catk8s-dashborad-service.yaml#-------------------DashboardService-------------------#kind:ServiceapiVersion:v1metadata: labels: k8s-app:kubernetes-dashboard name:kubernetes-dashboard namespace:kube-systemspec: ports: -port:443 targetPort:8443 nodePort:8490 type:NodePort selector: k8s-app:kubernetes-dashboard三、通过配置文件创建dashboard
#kubectlcreate-f.#kubectlgetpod,deployment,svc-nkube-system四、配置使用basic认证方式
默认情况下只支持kubeconfig和令牌认证
#echo'admin,admin,1'>/etc/kubernetes/basic_auth_file#grep'auth'/usr/lib/systemd/system/kube-apiserver.service --authorization-mode=Node,RBAC\\ --runtime-config=rbac.authorization.k8s.io/v1alpha1\\ --enable-bootstrap-token-auth=true\\ --token-auth-file=/etc/kubernetes/token.csv\\ --basic-auth-file=/etc/kubernetes/basic_auth_file\\#grep‘basic’k8s-dashborad-deployment.yaml（配置在args下面） ---authentication-mode=basic#systemctldaemon-reload#systemctlrestartkube-apiserver#kubectlapply-fk8s-dashborad-deployment.yaml
将admin用户和cluter-admin role进行角色绑定
#curl--insecurehttps://vm1:6443-basic-uadmin:admin#kubectlcreateclusterrolebinding\\login-on-dashboard-with-cluster-admin\\ --clusterrole=cluster-admin--user=admin#curl--insecurehttps://vm1:6443-basic-uadmin:admin五、访问测试六、整合heapster和influxdb
在没有配置heapster和influxdb的情况下，pod的metric信息是无法获取到的，而早前版本K8S的HPA特性依赖的metric数据来源恰巧就是heapster和influxdb。
1、准备yaml配置文件
#catheapster-sa.yamlapiVersion:v1kind:ServiceAccountmetadata: name:heapster namespace:kube-system
#catheapster-rbac.yamlkind:ClusterRoleBindingapiVersion:rbac.authorization.k8s.io/v1beta1metadata: name:heapsterroleRef: apiGroup:rbac.authorization.k8s.io kind:ClusterRole name:system:heapstersubjects:-kind:ServiceAccount name:heapster namespace:kube-system
#catheapster-deployment.yamlapiVersion:extensions/v1beta1 kind:Deployment metadata: name:heapsternamespace:kube-system spec: replicas:1 template: metadata: labels: task:monitoring k8s-app:heapster spec: serviceAccountName:heapster containers: -name:heapster image:k8s.gcr.io/heapster-amd64:v1.4.2 imagePullPolicy:IfNotPresent command: -/heapster ---source=kubernetes:https://kubernetes.default ---sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
#catheapster-service.yamlapiVersion:v1kind:Servicemetadata: labels: task:monitoring kubernetes.io/cluster-service:'true' kubernetes.io/name:Heapster name:heapster namespace:kube-systemspec: ports: -port:80 targetPort:8082 selector:k8s-app:heapster
#catinfluxdb-deployment.yamlapiVersion:extensions/v1beta1kind:Deploymentmetadata:
name:monitoring-influxdb
namespace:kube-systemspec:
replicas:1
template:
metadata:
labels:
task:monitoring
k8s-app:influxdb
spec:
containers:
-name:influxdb
image:k8s.gcr.io/heapster-influxdb-amd64:v1.3.3
volumeMounts:
-mountPath:/data
name:influxdb-storage
volumes:
-name:influxdb-storage
emptyDir:{}#catinfluxdb-service.yamlapiVersion:v1kind:Servicemetadata: labels: task:monitoring kubernetes.io/cluster-service:'true' kubernetes.io/name:monitoring-influxdb name:monitoring-influxdb namespace:kube-systemspec: ports: -port:8086 targetPort:8086 selector: k8s-app:influxdb
获取heapster中的获取支持的metrics
#kubectlrun-i--ttycurl--namespace=kube-system\\--image=registry.59iedu.com/webwurst/curl-utils/bin/sh #curlhttp://heapster/api/v1/model/metrics#curlhttp://heapster/api/v1/model/debug/allkeys
#kubectlgetnode#kubectltopnode
当heapster和influxdb pod都正常运行的时候，在dashboard里面就可以看到CPU和内存的监控数据了。
1、删除apiserver里面basic认证相关的配置后重启apiserver
--basic-auth-file=/etc/kubernetes/basic_auth_file
#systemctldaemon-reload#systemctlrestartkube-apiserver
2、删除clusterrolebinding
#kubectldeleteclusterrolebindinglogin-on-dashboard-with-cluster-admin
3、修改k8s-dashborad-deployment.yaml文件
去掉- --authentication-mode=basic参数
4、创建普通用户,赋予所有namespace下资源的get、watch和list权限。
这里通过clusterrole和culsterrolebinding赋予所有namespace相关资源的get、watch、list权限，实际应用环境建议使用创建role和rolebinding指定特定的namespace相关资源权限，各资源权限的赋予规则遵循最小权限原则。
#catrbac-yang.yamlkind:ClusterRoleapiVersion:rbac.authorization.k8s.io/v1metadata: name:role-yangrules:-apiGroups:[""] resources:["*"] verbs:["get","watch","list"] -apiGroups:["storage.k8s.io"] resources:["*"] verbs:["get","watch","list"] -apiGroups:["rbac.authorization.k8s.io"] resources:["*"] verbs:["get","watch","list"] -apiGroups:["batch"] resources:["*"] verbs:["get","watch","list"] -apiGroups:["apps"] resources:["*"] verbs:["get","watch","list"] -apiGroups:["extensions"] resources:["*"] verbs:["get","watch","list"] ---kind:ClusterRoleBindingapiVersion:rbac.authorization.k8s.io/v1metadata: name:role-bind-yangsubjects:-kind:ServiceAccount name:yang namespace:kube-systemroleRef: kind:ClusterRole name:role-yang apiGroup:rbac.authorization.k8s.io
#kubectlcreatesayang-nkube-system#kubectlcreate-frbac-yang.yaml#kubectl-nkube-systemdescribesecret$(kubectl-nkube-systemgetsecret|grepyang|awk'{print$1}')5、测试普通用户的权限
6、创建super用户admin
#kubectlcreatesaadmin-nkube-system#catrbac-admin.yamlapiVersion:rbac.authorization.k8s.io/v1beta1
kind:ClusterRoleBinding
metadata:
name:admin
roleRef:
apiGroup:rbac.authorization.k8s.io
kind:ClusterRole
name:cluster-admin
subjects:
-kind:ServiceAccount
name:admin
namespace:kube-system#kubectlcreate-frbac-admin.yaml#kubectl-nkube-systemdescribesecret$(kubectl-nkube-systemgetsecret|grepadmin|awk'{print$1}')
使用admin用户的token登陆后继承cluster-admin的权限
参考：
https://github.com/kubernetes/dashboard/wiki/Creating-sample-user
https://github.com/kubernetes/dashboard/wiki/Access-control
https://github.com/kubernetes/heapster/blob/master/docs/model.md
