通过在Service的Yaml文件中配置注解Annotation，您可以对天翼云负载均衡、监听转发等进行配置，本文介绍LoadBalancer类型Service支持配置的Annotation。

注解说明

健康检查字段数据结构说明

使用已有负载均衡

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"  #负载均衡ELB实例的ID
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

使用已有负载均衡，并强制覆盖已有监听

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"  #负载均衡ELB实例的ID
    service.beta.kubernetes.io/ctyun-loadbalancer-override-listeners: "true"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

创建私网负载均衡（包年包月）

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-spec: "elb.s2.small"  #新建负载均衡规格为标准型Ⅰ
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type: "intranet"  #新建负载均衡为私网类型
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-count: "1" # 新建负载均衡计费周期为1个月
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

创建私网负载均衡（按需计费）

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-spec: "elb.s2.small"  #新建负载均衡规格为标准型Ⅰ
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type: "intranet"  #新建负载均衡为私网类型
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type: "on_demand" # 新建负载均衡计费模式为按需计费
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

新建公网负载均衡(包年包月）

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-spec: "elb.s2.small"  #新建负载均衡规格为标准型Ⅰ
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type: "internet"  #新建负载均衡为公网类型
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-count: "1" # 新建负载均衡计费周期为1个月
    service.beta.kubernetes.io/ctyun-loadbalancer-charge-type: "bandwidth"  #公网计费按带宽计费
    service.beta.kubernetes.io/ctyun-loadbalancer-bandwidth: "5"  #公网带宽大小为5Mbps
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

新建公网负载均衡(按需计费）

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-spec: "elb.s2.small"  #新建负载均衡规格为标准型Ⅰ
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type: "internet"  #新建负载均衡为公网类型
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type: "on_demand" # 新建负载均衡计费模式为按需计费
    service.beta.kubernetes.io/ctyun-loadbalancer-charge-type: "bandwidth"  #公网计费按带宽计费
    service.beta.kubernetes.io/ctyun-loadbalancer-bandwidth: "5"  #公网带宽大小为5Mbps
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

新建IPv6类型负载均衡

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type: "internet"
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type: "on_demand"
    service.beta.kubernetes.io/ctyun-loadbalancer-enable-ipv6: "true"
    service.beta.kubernetes.io/ctyun-loadbalancer-enable-listener-nat64: "true"
    service.beta.kubernetes.io/ctyun-loadbalancer-ipv6-bandwidth-id: "bandwidth-ylupxfrf8i"
    service.beta.kubernetes.io/ctyun-loadbalancer-spec: "elb.s2.small"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

删除Service时保留自动创建的负载均衡实例

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-spec: "elb.s2.small"  #新建负载均衡规格为标准型Ⅰ
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type: "intranet"  #新建负载均衡为私网类型
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type: "on_demand" # 新建负载均衡计费模式为按需计费  
    service.beta.kubernetes.io/ctyun-loadbalancer-reserved: "true"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

配置监听HTTP协议

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-protocol-port: "http:80"  #监听HTTP的80端口
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

配置监听HTTPS协议

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-protocol-port: "https:443"  #监听HTTPS的443端口
    service.beta.kubernetes.io/ctyun-loadbalancer-ssl-cert: "${YOUR_CERT_ID}"  #SSL证书ID
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 443
      targetPort: 80
  type: LoadBalancer

配置附加X-Forwarded-For请求头

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-xforwardedfor: "true"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

配置全局健康检查

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/ctyun-loadbalancer-health-check-option: '{
      "protocol":"TCP",
      "interval":"5",
      "timeout":"10",
      "maxRetry":"3"
    }'
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
    - protocol: TCP
      port: 90
      targetPort: 90
  type: LoadBalancer

配置部分端口健康检查

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/ctyun-loadbalancer-health-check-options: '[
      {
        "servicePort":"TCP:90",  #指定监听协议为TCP、服务端口为90的端口配置
        "protocol":"TCP",        #指定健康检查协议为TCP
        "interval":"5",
        "timeout":"10",
        "maxRetry":"3"
      },
      {
        "servicePort":"TCP:80",  #指定监听协议为TCP、服务端口为80的端口配置
        "protocol":"HTTP",       #指定健康检查协议为HTTP
        "interval":"5",
        "timeout":"10",
        "maxRetry":"3",
        "path":"/healthz",
        "expectedCodes":"http_2xx"
      },
    ]'
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
    - protocol: TCP
      port: 90
      targetPort: 90
  type: LoadBalancer

配置访问控制黑/白名单

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-acl-flag: "white"         #白名单控制
    service.beta.kubernetes.io/ctyun-loadbalancer-acl-status: "on"          #开启访问控制
    service.beta.kubernetes.io/ctyun-loadbalancer-acl-id: "${YOUR_ACL_ID}"  #ELB的访问策略组ID
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

为TCP监听配置Proxy Protocol协议

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-proxy-protocol-flag: "on"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

开启客户端源地址保持 

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-preserve-source-ip: "true"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  externalTrafficPolicy: Local

为监听设置连接空闲超时时间

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/ctyun-loadbalancer-idle-timeout: "30"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

为监听设置响应超时时间

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/ctyun-loadbalancer-response-timeout: "30"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

为监听设置建立连接超时时间

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-establish-timeout: "30"
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

指定后端主机组调度算法

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-scheduler: "wrr"
spec:
  selector:
    app: nginx
  externalTrafficPolicy: "Local"    
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

开启修改/删除保护

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type:"intranet"
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type:"on_demand"
    service.beta.kubernetes.io/ctyun-loadbalancer-spec:"elb.s2.small"
    service.beta.kubernetes.io/ctyun-loadbalancer-delete-protection:"on"
    service.beta.kubernetes.io/ctyun-loadbalancer-modification-protection:"consoleProtection"
spec:
  selector:
    app: nginx   
  ports:
- protocol: TCP
      port:8080
      targetPort:8080
  type: LoadBalancer

使用指定Label的Worker节点作为后端服务器

apiVersion:"v1"
kind:"Service"
metadata:
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type:"on_demand"
    service.beta.kubernetes.io/ctyun-loadbalancer-scheduler:"wrr"
    service.beta.kubernetes.io/ctyun-loadbalancer-spec:"elb.s2.small"
    service.beta.kubernetes.io/ctyun-loadbalancer-backend-label:"test=test"
  name:"test-backend-label"
  namespace:"default"
spec:
  externalTrafficPolicy:"Cluster"
  ports:
  - name:"test"
    port:8080
    protocol:"TCP"
    targetPort:80
  selector:
    app:"test"
    version:"v1"
  sessionAffinity:"None"

移除ELB后端unschedulable状态的节点

apiVersion:"v1"
kind:"Service"
metadata:
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-acl-flag:"inherit"
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type:"intranet"
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type:"on_demand"
    service.beta.kubernetes.io/ctyun-loadbalancer-health-check-flag:"off"
    service.beta.kubernetes.io/ctyun-loadbalancer-scheduler:"wrr"
    service.beta.kubernetes.io/ctyun-loadbalancer-spec:"elb.s2.small"
    service.beta.kubernetes.io/ctyun-loadbalancer-remove-unscheduled-backend:"true"
  name:"test-remove-unscheduled-backend"
  namespace:"default"
spec:
  externalTrafficPolicy:"Cluster"
  ports:
  - name:"test"
    nodePort:31267
    port:7888
    protocol:"TCP"
    targetPort:80
  selector:
    app:"test"
    version:"v1"
  sessionAffinity:"None"
  type:"LoadBalancer"

指定后端主机组会话保持设置

示例1:http/https监听会话保持设置

apiVersion:"v1"
kind:"Service"
metadata:
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type:"on_demand"
    service.beta.kubernetes.io/ctyun-loadbalancer-protocol-port:"http:8080"
    service.beta.kubernetes.io/ctyun-loadbalancer-scheduler:"wrr"
    service.beta.kubernetes.io/ctyun-loadbalancer-session-sticky-cookie-expire:"10"
    service.beta.kubernetes.io/ctyun-loadbalancer-session-sticky-mode:"INSERT"
    service.beta.kubernetes.io/ctyun-loadbalancer-spec:"elb.s2.small"
  name:"test-session-sticky-http"
  namespace:"default"
spec:
  externalTrafficPolicy:"Local"
  ports:
  - name:"test"
    port:8080
    protocol:"TCP"
    targetPort:80
  selector:
    app:"test"
    version:"v1"
  sessionAffinity:"None"
  type:"LoadBalancer"

示例2:tcp/udp监听会话保持设置

apiVersion:"v1"
kind:"Service"
metadata:
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type:"on_demand"
    service.beta.kubernetes.io/ctyun-loadbalancer-scheduler:"wrr"
    service.beta.kubernetes.io/ctyun-loadbalancer-session-sticky-mode:"SOURCE_IP"
    service.beta.kubernetes.io/ctyun-loadbalancer-session-sticky-sourceip-timeout:"10"
    service.beta.kubernetes.io/ctyun-loadbalancer-spec:"elb.s2.small"
  name:"test-session-sticky-tcp"
  namespace:"default"
spec:
  externalTrafficPolicy:"Local"
  ports:
  - name:"test"
    port:8080
    protocol:"TCP"
    targetPort:80
  selector:
    app:"test"
    version:"v1"
  sessionAffinity:"None"
  type:"LoadBalancer"

后端主机组设置优雅中断

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-spec:"elb.s2.small"
    service.beta.kubernetes.io/ctyun-loadbalancer-address-type:"intranet"
    service.beta.kubernetes.io/ctyun-loadbalancer-cycle-type:"on_demand"
    service.beta.kubernetes.io/ctyun-loadbalancer-scheduler:"wrr"
    service.beta.kubernetes.io/ctyun-loadbalancer-connection-drain:"on"
    service.beta.kubernetes.io/ctyun-loadbalancer-connection-drain-timeout-seconds:"100"
spec:
  selector:
    app: nginx
  externalTrafficPolicy:"Local"
  ports:
- protocol: TCP
      port:80
      targetPort:80
  type: LoadBalancer

复用已有后端主机组

kind: Service
apiVersion: v1
metadata:
  name: nginx
  annotations:
    service.beta.kubernetes.io/ctyun-loadbalancer-id:"lb-xxxx"
    service.beta.kubernetes.io/ctyun-loadbalancer-target-group-port:"sgp-1eqlll02zooys3op1j:8080"
    service.beta.kubernetes.io/ctyun-loadbalancerweight:"100"
spec:
  selector:
    app: nginx   
  ports:
- protocol: TCP
      port:8080
      targetPort:80
  type: LoadBalancer