基于CCE One集群联邦进行多集群应用分发与管理-天翼云开发者社区

CCE One集群联邦支持通过PropagationPolicy定制资源模板分发策略，以及支持通过OverridePolicy来定制资源模板分发到不同成员集群的差异化配置。

这里，我们以分发一个Nginx Deployment Web应用为例，简要介绍如何实现应用的多集群分发。

前提条件

已开通CCE One集群联邦实例，可按需给apiserver绑定公网EIP以暴露到公网访问；
通过集群联邦接入配置，获取到联邦apiserver访问的KubeConfig文件；
联邦中有添加成员集群，且成员集群状态为Ready（代表已联通）；

通过kubectl命令查询联邦控制面中成员集群列表及状态，可参考如下命令：

# 将如下‘/home/ctyun/.kube/cceone-fed.yaml’替换为您本地的联邦KubeConfig文件路径
# 集群状态Ready=True代表已正常接入，可进行应用分发

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get cluster
NAME               VERSION   MODE   READY   AGE
cce-cceone-demo1   v1.27.8   Push   True    16d
cce-cceone-demo2   v1.22.1   Push   True    56d
cceone-c2          v1.22.1   Push   True    55d
cceone-cluster1    v1.22.1   Push   True    55d
icce-yhn-296                 Push   False   16d

操作指引

该操作指引以分发一个nginx deployment到成员集群为例，若需分发其他资源（包括CR），只需将nginx deployment替换为具体的资源模板即可。

需要注意的是：

资源模板和PropagationPolicy之间可任意创建顺序不分先后，当二者匹配后即立即触发调度分发逻辑；
差异化策略需要在触发调度分发逻辑之前创建好，也即差异化策略不能是最后一个；

如下为详细操作参考：

步骤一：创建资源模板

执行如下命令，在联邦控制面中创建资源模板：

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml create deployment nginx --image nginx --replicas=3
deployment.apps/nginx created

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/3     0            0           21s

此时，只是将K8S YAML apply到联邦控制面，还未分发到成员集群；联邦控制面当前也并不会调谐这些workload，也即实际看到的READY副本数会为0；

步骤二：创建差异化策略

差异化策略(OverridePolicy)模板参考如下。若不需要做集群间差异化，则可以略过该步骤。

# overridepolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
  overrideRules:
    - targetCluster:
        clusterNames:
          - cce-cceone-demo1
      overriders:
        labelsOverrider:
          - operator: add
            value:
              cceone.propagationto: cce-cceone-demo1
    - targetCluster:
        clusterNames:
          - cce-cceone-demo2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              cceone.propagationto: cce-cceone-demo2

将如上YAML文件通过如下命令apply到联邦控制面中：

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f overridepolicy.yaml 
overridepolicy.policy.karmada.io/example created
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get overridepolicy example
NAME      AGE
example   21s

此时，通过命令查看nginx应用资源模板，还是未分发状态

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/3     0            0           61m

步骤三：创建调度策略

调度策略（PropagationPolicy）模板参考如下，可基于实际需要调整其中的污点容忍及副本拆分策略等：

# propagationpolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-policy # The default namespace is `default`.
spec:
  propagateDeps: true
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
  placement:
    clusterAffinity:    # 指定目标集群
      clusterNames:            
        - cce-cceone-demo1
        - cce-cceone-demo2
    clusterTolerations:  # 容忍成员集群污点，影响调度逻辑
      - key: cluster.karmada.io/not-ready       
        operator: Exists          
        effect: NoExecute          
        tolerationSeconds: 300
      - key: cluster.karmada.io/unreachable     
        operator: Exists          
        effect: NoExecute          
        tolerationSeconds: 300
    replicaScheduling:  # 指定应用副本在成员集群间的分发策略
      replicaSchedulingType: Divided        
      replicaDivisionPreference: Weighted        
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - cce-cceone-demo1            
            weight: 1                          
          - targetCluster:
              clusterNames:
                - cce-cceone-demo2                
            weight: 2

通过如下命令，将调度策略apply到联邦控制面中

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f propagationpolicy.yaml 
propagationpolicy.policy.karmada.io/example-policy created
ctyun@0000000g-FxXsKxFWUv:~/tmp$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get propagationpolicy example-policy
NAME             CONFLICT-RESOLUTION   PRIORITY   AGE
example-policy   Abort                 0          13s

此时，通过如下命令，查看应用在多集群之间的调度情况：

# 查看应用成员Pod Ready情况
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   3/3     3            3           10s
# 查询应用多集群调度结果
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get resourcebinding nginx-deployment -o yaml
apiVersion: work.karmada.io/v1alpha2
kind: ResourceBinding
metadata:
  annotations:
    policy.karmada.io/applied-placement: '{"clusterAffinity":{"clusterNames":["cce-cceone-demo1","cce-cceone-demo2"]},"clusterTolerations":[{"key":"cluster.karmada.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"cluster.karmada.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"replicaScheduling":{"replicaSchedulingType":"Divided","replicaDivisionPreference":"Weighted","weightPreference":{"staticWeightList":[{"targetCluster":{"clusterNames":["cce-cceone-demo1"]},"weight":1},{"targetCluster":{"clusterNames":["cce-cceone-demo2"]},"weight":2}]}}}'
    propagationpolicy.karmada.io/name: example-policy
    propagationpolicy.karmada.io/namespace: default
    resourcebinding.karmada.io/dependencies: "null"
  creationTimestamp: "2025-05-30T09:52:51Z"
  finalizers:
  - karmada.io/binding-controller
  - karmada.io/binding-dependencies-distributor
  generation: 2
  labels:
    propagationpolicy.karmada.io/permanent-id: bb4b469a-6c93-4ed7-8465-e91564eb33eb
    resourcebinding.karmada.io/permanent-id: 89904db8-95a4-4820-b3b7-cea95b1c3e51
  name: nginx-deployment
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: nginx
    uid: 86da5387-b36b-40c4-99a2-462cbc832c99
  resourceVersion: "44338541"
  uid: 90b31326-7a74-405e-81e3-98ac18969d9f
spec:
  clusters:
  - name: cce-cceone-demo2
    replicas: 2
  - name: cce-cceone-demo1
    replicas: 1
  conflictResolution: Abort
  placement:
    clusterAffinity:
      clusterNames:
      - cce-cceone-demo1
      - cce-cceone-demo2
    clusterTolerations:
    - effect: NoExecute
      key: cluster.karmada.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: cluster.karmada.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
        - targetCluster:
            clusterNames:
            - cce-cceone-demo1
          weight: 1
        - targetCluster:
            clusterNames:
            - cce-cceone-demo2
          weight: 2
  propagateDeps: true
  replicas: 3
  resource:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
    namespace: default
    resourceVersion: "44338446"
    uid: 86da5387-b36b-40c4-99a2-462cbc832c99
  schedulerName: default-scheduler
status:
  aggregatedStatus:      # 此处为聚合了成员集群中该Deployment对应副本Pod的部署状态
  - applied: true
    clusterName: cce-cceone-demo1
    health: Healthy
    status:
      availableReplicas: 1
      generation: 1
      observedGeneration: 1
      readyReplicas: 1
      replicas: 1
      resourceTemplateGeneration: 2
      updatedReplicas: 1
  - applied: true
    clusterName: cce-cceone-demo2
    health: Healthy
    status:
      availableReplicas: 2
      generation: 1
      observedGeneration: 1
      readyReplicas: 2
      replicas: 2
      resourceTemplateGeneration: 2
      updatedReplicas: 2
  conditions:
  - lastTransitionTime: "2025-05-30T09:52:51Z"
    message: Binding has been scheduled successfully.
    reason: Success
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2025-05-30T09:52:51Z"
    message: All works have been successfully applied
    reason: FullyAppliedSuccess
    status: "True"
    type: FullyApplied
  lastScheduledTime: "2025-05-30T09:52:51Z"
  schedulerObservedGeneration: 2

CCE One集群联邦支持通过PropagationPolicy定制资源模板分发策略，以及支持通过OverridePolicy来定制资源模板分发到不同成员集群的差异化配置。

这里，我们以分发一个Nginx Deployment Web应用为例，简要介绍如何实现应用的多集群分发。

前提条件

已开通CCE One集群联邦实例，可按需给apiserver绑定公网EIP以暴露到公网访问；
通过集群联邦接入配置，获取到联邦apiserver访问的KubeConfig文件；
联邦中有添加成员集群，且成员集群状态为Ready（代表已联通）；

通过kubectl命令查询联邦控制面中成员集群列表及状态，可参考如下命令：

# 将如下‘/home/ctyun/.kube/cceone-fed.yaml’替换为您本地的联邦KubeConfig文件路径
# 集群状态Ready=True代表已正常接入，可进行应用分发

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get cluster
NAME               VERSION   MODE   READY   AGE
cce-cceone-demo1   v1.27.8   Push   True    16d
cce-cceone-demo2   v1.22.1   Push   True    56d
cceone-c2          v1.22.1   Push   True    55d
cceone-cluster1    v1.22.1   Push   True    55d
icce-yhn-296                 Push   False   16d

操作指引

该操作指引以分发一个nginx deployment到成员集群为例，若需分发其他资源（包括CR），只需将nginx deployment替换为具体的资源模板即可。

需要注意的是：

资源模板和PropagationPolicy之间可任意创建顺序不分先后，当二者匹配后即立即触发调度分发逻辑；
差异化策略需要在触发调度分发逻辑之前创建好，也即差异化策略不能是最后一个；

如下为详细操作参考：

步骤一：创建资源模板

执行如下命令，在联邦控制面中创建资源模板：

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml create deployment nginx --image nginx --replicas=3
deployment.apps/nginx created

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/3     0            0           21s

此时，只是将K8S YAML apply到联邦控制面，还未分发到成员集群；联邦控制面当前也并不会调谐这些workload，也即实际看到的READY副本数会为0；

步骤二：创建差异化策略

差异化策略(OverridePolicy)模板参考如下。若不需要做集群间差异化，则可以略过该步骤。

# overridepolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
  overrideRules:
    - targetCluster:
        clusterNames:
          - cce-cceone-demo1
      overriders:
        labelsOverrider:
          - operator: add
            value:
              cceone.propagationto: cce-cceone-demo1
    - targetCluster:
        clusterNames:
          - cce-cceone-demo2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              cceone.propagationto: cce-cceone-demo2

将如上YAML文件通过如下命令apply到联邦控制面中：

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f overridepolicy.yaml 
overridepolicy.policy.karmada.io/example created
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get overridepolicy example
NAME      AGE
example   21s

此时，通过命令查看nginx应用资源模板，还是未分发状态

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/3     0            0           61m

步骤三：创建调度策略

调度策略（PropagationPolicy）模板参考如下，可基于实际需要调整其中的污点容忍及副本拆分策略等：

# propagationpolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-policy # The default namespace is `default`.
spec:
  propagateDeps: true
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
  placement:
    clusterAffinity:    # 指定目标集群
      clusterNames:            
        - cce-cceone-demo1
        - cce-cceone-demo2
    clusterTolerations:  # 容忍成员集群污点，影响调度逻辑
      - key: cluster.karmada.io/not-ready       
        operator: Exists          
        effect: NoExecute          
        tolerationSeconds: 300
      - key: cluster.karmada.io/unreachable     
        operator: Exists          
        effect: NoExecute          
        tolerationSeconds: 300
    replicaScheduling:  # 指定应用副本在成员集群间的分发策略
      replicaSchedulingType: Divided        
      replicaDivisionPreference: Weighted        
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - cce-cceone-demo1            
            weight: 1                          
          - targetCluster:
              clusterNames:
                - cce-cceone-demo2                
            weight: 2

通过如下命令，将调度策略apply到联邦控制面中

$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f propagationpolicy.yaml 
propagationpolicy.policy.karmada.io/example-policy created
ctyun@0000000g-FxXsKxFWUv:~/tmp$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get propagationpolicy example-policy
NAME             CONFLICT-RESOLUTION   PRIORITY   AGE
example-policy   Abort                 0          13s

此时，通过如下命令，查看应用在多集群之间的调度情况：

# 查看应用成员Pod Ready情况
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx 
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   3/3     3            3           10s
# 查询应用多集群调度结果
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get resourcebinding nginx-deployment -o yaml
apiVersion: work.karmada.io/v1alpha2
kind: ResourceBinding
metadata:
  annotations:
    policy.karmada.io/applied-placement: '{"clusterAffinity":{"clusterNames":["cce-cceone-demo1","cce-cceone-demo2"]},"clusterTolerations":[{"key":"cluster.karmada.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"cluster.karmada.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"replicaScheduling":{"replicaSchedulingType":"Divided","replicaDivisionPreference":"Weighted","weightPreference":{"staticWeightList":[{"targetCluster":{"clusterNames":["cce-cceone-demo1"]},"weight":1},{"targetCluster":{"clusterNames":["cce-cceone-demo2"]},"weight":2}]}}}'
    propagationpolicy.karmada.io/name: example-policy
    propagationpolicy.karmada.io/namespace: default
    resourcebinding.karmada.io/dependencies: "null"
  creationTimestamp: "2025-05-30T09:52:51Z"
  finalizers:
  - karmada.io/binding-controller
  - karmada.io/binding-dependencies-distributor
  generation: 2
  labels:
    propagationpolicy.karmada.io/permanent-id: bb4b469a-6c93-4ed7-8465-e91564eb33eb
    resourcebinding.karmada.io/permanent-id: 89904db8-95a4-4820-b3b7-cea95b1c3e51
  name: nginx-deployment
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: nginx
    uid: 86da5387-b36b-40c4-99a2-462cbc832c99
  resourceVersion: "44338541"
  uid: 90b31326-7a74-405e-81e3-98ac18969d9f
spec:
  clusters:
  - name: cce-cceone-demo2
    replicas: 2
  - name: cce-cceone-demo1
    replicas: 1
  conflictResolution: Abort
  placement:
    clusterAffinity:
      clusterNames:
      - cce-cceone-demo1
      - cce-cceone-demo2
    clusterTolerations:
    - effect: NoExecute
      key: cluster.karmada.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: cluster.karmada.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
        - targetCluster:
            clusterNames:
            - cce-cceone-demo1
          weight: 1
        - targetCluster:
            clusterNames:
            - cce-cceone-demo2
          weight: 2
  propagateDeps: true
  replicas: 3
  resource:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
    namespace: default
    resourceVersion: "44338446"
    uid: 86da5387-b36b-40c4-99a2-462cbc832c99
  schedulerName: default-scheduler
status:
  aggregatedStatus:      # 此处为聚合了成员集群中该Deployment对应副本Pod的部署状态
  - applied: true
    clusterName: cce-cceone-demo1
    health: Healthy
    status:
      availableReplicas: 1
      generation: 1
      observedGeneration: 1
      readyReplicas: 1
      replicas: 1
      resourceTemplateGeneration: 2
      updatedReplicas: 1
  - applied: true
    clusterName: cce-cceone-demo2
    health: Healthy
    status:
      availableReplicas: 2
      generation: 1
      observedGeneration: 1
      readyReplicas: 2
      replicas: 2
      resourceTemplateGeneration: 2
      updatedReplicas: 2
  conditions:
  - lastTransitionTime: "2025-05-30T09:52:51Z"
    message: Binding has been scheduled successfully.
    reason: Success
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2025-05-30T09:52:51Z"
    message: All works have been successfully applied
    reason: FullyAppliedSuccess
    status: "True"
    type: FullyApplied
  lastScheduledTime: "2025-05-30T09:52:51Z"
  schedulerObservedGeneration: 2

智算服务

应用商城

合作伙伴

开发者

支持与服务

了解天翼云

基于CCE One集群联邦进行多集群应用分发与管理

前提条件

操作指引

基于CCE One集群联邦进行多集群应用分发与管理

前提条件

操作指引

活动

智算服务

应用商城

合作伙伴

开发者

支持与服务

了解天翼云

基于CCE One集群联邦进行多集群应用分发与管理

前提条件

操作指引

基于CCE One集群联邦进行多集群应用分发与管理

前提条件

操作指引