CCE One集群联邦支持通过PropagationPolicy定制资源模板分发策略,以及支持通过OverridePolicy来定制资源模板分发到不同成员集群的差异化配置。
这里,我们以分发一个Nginx Deployment Web应用为例,简要介绍如何实现工作负载的多集群应用分发。
前提条件
已开通CCE One集群联邦实例,可按需给apiserver绑定公网EIP以暴露到公网访问。
通过集群联邦接入配置,获取到联邦apiserver访问的KubeConfig文件。
联邦中有添加成员集群,且成员集群状态为Ready(代表已联通)。
通过kubectl命令查询联邦控制面中成员集群列表及状态,可参考如下命令:
# 将如下‘/home/ctyun/.kube/cceone-fed.yaml’替换为您本地的联邦KubeConfig文件路径
# 集群状态Ready=True代表已正常接入,可进行应用分发
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get cluster
NAME VERSION MODE READY AGE
cce-cceone-demo1 v1.27.8 Push True 16d
cce-cceone-demo2 v1.22.1 Push True 56d
cceone-c2 v1.22.1 Push True 55d
cceone-cluster1 v1.22.1 Push True 55d
icce-yhn-296 Push False 16d
操作指引
该操作指引以分发一个nginx deployment到成员集群为例,若需分发其他资源(包括CR),只需将nginx deployment替换为具体的资源模板即可。
注意
资源模板和PropagationPolicy之间可任意创建顺序不分先后,当二者匹配后即立即触发调度分发逻辑。
差异化策略需要在触发调度分发逻辑之前创建好,也即差异化策略不能是最后一个。
如下为详细操作参考:
步骤一: 创建资源模板
执行如下命令,在联邦控制面中创建资源模板:
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml create deployment nginx --image nginx --replicas=3
deployment.apps/nginx created
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/3 0 0 21s
此时,只是将K8S YAML apply到联邦控制面,还未分发到成员集群;联邦控制面当前也并不会调谐这些workload,也即实际看到的READY副本数会为0;
步骤二:创建差异化策略
差异化策略(OverridePolicy)模板参考如下。若不需要做集群间差异化,则可以略过该步骤。
# overridepolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
name: example
spec:
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
overrideRules:
- targetCluster:
clusterNames:
- cce-cceone-demo1
overriders:
labelsOverrider:
- operator: add
value:
cceone.propagationto: cce-cceone-demo1
- targetCluster:
clusterNames:
- cce-cceone-demo2
overriders:
labelsOverrider:
- operator: add
value:
cceone.propagationto: cce-cceone-demo2
将如上YAML文件通过如下命令apply到联邦控制面中:
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f overridepolicy.yaml
overridepolicy.policy.karmada.io/example created
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get overridepolicy example
NAME AGE
example 21s
此时,通过命令查看nginx工作负载,还是未分发状态
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/3 0 0 61m
步骤三:创建调度策略
调度策略(PropagationPolicy)模板参考如下,可基于实际需要调整其中的污点容忍及负载拆分策略等:
# propagationpolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: example-policy # The default namespace is `default`.
spec:
propagateDeps: true
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
placement:
clusterAffinity: # 指定目标集群
clusterNames:
- cce-cceone-demo1
- cce-cceone-demo2
clusterTolerations: # 容忍成员集群污点,影响调度逻辑
- key: cluster.karmada.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 300
- key: cluster.karmada.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 300
replicaScheduling: # 指定工作负载副本在成员集群间的分发策略
replicaSchedulingType: Divided
replicaDivisionPreference: Weighted
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- cce-cceone-demo1
weight: 1
- targetCluster:
clusterNames:
- cce-cceone-demo2
weight: 2
通过如下命令,将调度策略apply到联邦控制面中
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f propagationpolicy.yaml
propagationpolicy.policy.karmada.io/example-policy created
ctyun@0000000g-FxXsKxFWUv:~/tmp$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get propagationpolicy example-policy
NAME CONFLICT-RESOLUTION PRIORITY AGE
example-policy Abort 0 13s
此时,通过如下命令,查看工作负载在多集群之间的调度情况:
# 查看工作负载成员Pod Ready情况
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/3 3 3 10s
# 查询工作负载多集群调度结果
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get resourcebinding nginx-deployment -o yaml
apiVersion: work.karmada.io/v1alpha2
kind: ResourceBinding
metadata:
annotations:
policy.karmada.io/applied-placement: '{"clusterAffinity":{"clusterNames":["cce-cceone-demo1","cce-cceone-demo2"]},"clusterTolerations":[{"key":"cluster.karmada.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"cluster.karmada.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"replicaScheduling":{"replicaSchedulingType":"Divided","replicaDivisionPreference":"Weighted","weightPreference":{"staticWeightList":[{"targetCluster":{"clusterNames":["cce-cceone-demo1"]},"weight":1},{"targetCluster":{"clusterNames":["cce-cceone-demo2"]},"weight":2}]}}}'
propagationpolicy.karmada.io/name: example-policy
propagationpolicy.karmada.io/namespace: default
resourcebinding.karmada.io/dependencies: "null"
creationTimestamp: "2025-05-30T09:52:51Z"
finalizers:
- karmada.io/binding-controller
- karmada.io/binding-dependencies-distributor
generation: 2
labels:
propagationpolicy.karmada.io/permanent-id: bb4b469a-6c93-4ed7-8465-e91564eb33eb
resourcebinding.karmada.io/permanent-id: 89904db8-95a4-4820-b3b7-cea95b1c3e51
name: nginx-deployment
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: Deployment
name: nginx
uid: 86da5387-b36b-40c4-99a2-462cbc832c99
resourceVersion: "44338541"
uid: 90b31326-7a74-405e-81e3-98ac18969d9f
spec:
clusters:
- name: cce-cceone-demo2
replicas: 2
- name: cce-cceone-demo1
replicas: 1
conflictResolution: Abort
placement:
clusterAffinity:
clusterNames:
- cce-cceone-demo1
- cce-cceone-demo2
clusterTolerations:
- effect: NoExecute
key: cluster.karmada.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: cluster.karmada.io/unreachable
operator: Exists
tolerationSeconds: 300
replicaScheduling:
replicaDivisionPreference: Weighted
replicaSchedulingType: Divided
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- cce-cceone-demo1
weight: 1
- targetCluster:
clusterNames:
- cce-cceone-demo2
weight: 2
propagateDeps: true
replicas: 3
resource:
apiVersion: apps/v1
kind: Deployment
name: nginx
namespace: default
resourceVersion: "44338446"
uid: 86da5387-b36b-40c4-99a2-462cbc832c99
schedulerName: default-scheduler
status:
aggregatedStatus: # 此处为聚合了成员集群中该Deployment对应副本Pod的部署状态
- applied: true
clusterName: cce-cceone-demo1
health: Healthy
status:
availableReplicas: 1
generation: 1
observedGeneration: 1
readyReplicas: 1
replicas: 1
resourceTemplateGeneration: 2
updatedReplicas: 1
- applied: true
clusterName: cce-cceone-demo2
health: Healthy
status:
availableReplicas: 2
generation: 1
observedGeneration: 1
readyReplicas: 2
replicas: 2
resourceTemplateGeneration: 2
updatedReplicas: 2
conditions:
- lastTransitionTime: "2025-05-30T09:52:51Z"
message: Binding has been scheduled successfully.
reason: Success
status: "True"
type: Scheduled
- lastTransitionTime: "2025-05-30T09:52:51Z"
message: All works have been successfully applied
reason: FullyAppliedSuccess
status: "True"
type: FullyApplied
lastScheduledTime: "2025-05-30T09:52:51Z"
schedulerObservedGeneration: 2