[Cilium] #5주차 (3) 클러스터 메쉬

25년도 Cilium Study 1기 정리 글입니다.

kind 클러스터 배포

# west
kind create cluster --name west --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000 # sample apps
    hostPort: 30000
  - containerPort: 30001 # hubble ui
    hostPort: 30001
- role: worker
  extraPortMappings:
  - containerPort: 30002 # sample apps
    hostPort: 30002
networking:
  podSubnet: "10.0.0.0/16"
  serviceSubnet: "10.2.0.0/16"
  disableDefaultCNI: true
  kubeProxyMode: none
EOF


docker exec -it west-control-plane sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
docker exec -it west-worker sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'

# East
kind create cluster --name east --image kindest/node:v1.33.2 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 31000 # sample apps
    hostPort: 31000
  - containerPort: 31001 # hubble ui
    hostPort: 31001
- role: worker
  extraPortMappings:
  - containerPort: 31002 # sample apps
    hostPort: 31002
networking:
  podSubnet: "10.1.0.0/16"
  serviceSubnet: "10.3.0.0/16"
  disableDefaultCNI: true
  kubeProxyMode: none
EOF


docker exec -it east-control-plane sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'
docker exec -it east-worker sh -c 'apt update && apt install tree psmisc lsof wget net-tools dnsutils tcpdump ngrep iputils-ping git -y'

k get po -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-674b8bbfcf-jb744                     0/1     Pending   0          3m
kube-system          coredns-674b8bbfcf-nkxpb                     0/1     Pending   0          3m
kube-system          etcd-west-control-plane                      1/1     Running   0          3m8s
kube-system          kube-apiserver-west-control-plane            1/1     Running   0          3m8s
kube-system          kube-controller-manager-west-control-plane   1/1     Running   0          3m8s
kube-system          kube-scheduler-west-control-plane            1/1     Running   0          3m8s
local-path-storage   local-path-provisioner-7dc846544d-k9klb      0/1     Pending   0          3m

아직 네트워크 cni가 안깔려있으므로 파드가 정상적으로 기동되지 않고 있다.

cilium 설치

brew install cilium-cli

cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.0.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.1.0.0/16}' \
--set cluster.name=west --set cluster.id=1 \
--context kind-west


cilium install --version 1.17.6 --set ipam.mode=kubernetes \
--set kubeProxyReplacement=true --set bpf.masquerade=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set operator.replicas=1 --set debug.enabled=true \
--set routingMode=native --set autoDirectNodeRoutes=true --set ipv4NativeRoutingCIDR=10.1.0.0/16 \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.0.0.0/16}' \
--set cluster.name=east --set cluster.id=2 \
--context kind-east

파드가 올라올 때까지 조금 시간이 걸리는데(한 5분?) 다 올라오고 나면 모든 파드들이 러닝 상태로 바뀐다.

상태 확인

kwest get pod -A && keast get pod -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          cilium-envoy-jfnpd                           1/1     Running   0          3m23s
kube-system          cilium-envoy-v8c45                           1/1     Running   0          3m23s
kube-system          cilium-gl54x                                 1/1     Running   0          3m23s
kube-system          cilium-operator-65c89f5685-h2kks             1/1     Running   0          3m23s
kube-system          cilium-w7vk9                                 1/1     Running   0          3m23s
kube-system          coredns-674b8bbfcf-jb744                     1/1     Running   0          37m
kube-system          coredns-674b8bbfcf-nkxpb                     1/1     Running   0          37m
kube-system          etcd-west-control-plane                      1/1     Running   0          37m
kube-system          kube-apiserver-west-control-plane            1/1     Running   0          37m
kube-system          kube-controller-manager-west-control-plane   1/1     Running   0          37m
kube-system          kube-scheduler-west-control-plane            1/1     Running   0          37m
local-path-storage   local-path-provisioner-7dc846544d-k9klb      1/1     Running   0          37m
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          cilium-envoy-9xrss                           1/1     Running   0          2m59s
kube-system          cilium-envoy-npq4q                           1/1     Running   0          2m59s
kube-system          cilium-jmlgt                                 1/1     Running   0          2m59s
kube-system          cilium-operator-86cc56ccbb-zpdvn             1/1     Running   0          2m59s
kube-system          cilium-t5mpn                                 1/1     Running   0          2m59s
kube-system          coredns-674b8bbfcf-jcjgg                     1/1     Running   0          31m
kube-system          coredns-674b8bbfcf-xnx4g                     1/1     Running   0          31m
kube-system          etcd-east-control-plane                      1/1     Running   0          31m
kube-system          kube-apiserver-east-control-plane            1/1     Running   0          31m
kube-system          kube-controller-manager-east-control-plane   1/1     Running   0          31m
kube-system          kube-scheduler-east-control-plane            1/1     Running   0          31m
local-path-storage   local-path-provisioner-7dc846544d-fctpp      1/1     Running   0          31m

kwest -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf ipmasq list
IP PREFIX/ADDRESS
10.1.0.0/16
169.254.0.0/16


keast -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf ipmasq list
IP PREFIX/ADDRESS
10.0.0.0/16
169.254.0.0/16


kubectl describe cm -n kube-system coredns --context kind-west | grep kubernetes
    kubernetes cluster.local in-addr.arpa ip6.arpa {


kubectl describe cm -n kube-system coredns --context kind-west | grep kubernetes
    kubernetes cluster.local in-addr.arpa ip6.arpa {

라우팅 정보 확인

west-control-plane (docker IP: 192.168.97.3)

default via 192.168.97.1 dev eth0
10.0.0.0/24 via 10.0.0.235 dev cilium_host  src 10.0.0.235   # 로컬 PodCIDR
10.0.0.235 dev cilium_host scope link
10.0.1.0/24 via 192.168.97.2 dev eth0                          # 원격(워커) PodCIDR
192.168.97.0/24 dev eth0  src 192.168.97.3


로컬 PodCIDR: 10.0.0.0/24 (게이트웨이 = cilium_host 10.0.0.235)
원격 PodCIDR: 10.0.1.0/24 → west-worker(192.168.97.2) 로 전달

west-worker (docker IP: 192.168.97.2)

default via 192.168.97.1 dev eth0
10.0.0.0/24 via 192.168.97.3 dev eth0                          # 원격(컨트롤) PodCIDR
10.0.1.0/24 via 10.0.1.76 dev cilium_host  src 10.0.1.76       # 로컬 PodCIDR
10.0.1.76 dev cilium_host scope link
192.168.97.0/24 dev eth0  src 192.168.97.2


로컬 PodCIDR: 10.0.1.0/24 (게이트웨이 = cilium_host 10.0.1.76)
원격 PodCIDR: 10.0.0.0/24 → west-control-plane(192.168.97.3) 로 전달

east-control-plane (docker IP: 192.168.97.4)

default via 192.168.97.1 dev eth0
10.1.0.0/24 via 10.1.0.7 dev cilium_host  src 10.1.0.7         # 로컬 PodCIDR
10.1.0.7 dev cilium_host scope link
10.1.1.0/24 via 192.168.97.5 dev eth0                          # 원격(워커) PodCIDR
192.168.97.0/24 dev eth0  src 192.168.97.4


로컬 PodCIDR: 10.1.0.0/24 (게이트웨이 = cilium_host 10.1.0.7)
원격 PodCIDR: 10.1.1.0/24 → east-worker(192.168.97.5) 로 전달

east-worker (docker IP: 192.168.97.5)

default via 192.168.97.1 dev eth0
10.1.0.0/24 via 192.168.97.4 dev eth0                          # 원격(컨트롤) PodCIDR
10.1.1.0/24 via 10.1.1.73 dev cilium_host  src 10.1.1.73       # 로컬 PodCIDR
10.1.1.73 dev cilium_host scope link
192.168.97.0/24 dev eth0  src 192.168.97.5


로컬 PodCIDR: 10.1.1.0/24 (게이트웨이 = cilium_host 10.1.1.73)
원격 PodCIDR: 10.1.0.0/24 → east-control-plane(192.168.97.4) 로 전달

현재는 east ↔ west 클러스터 간 통신은 안 되는데, 그 이유는 각각의 노드들의 라우팅에 자기 클러스터 PodCIDR만 있고( west=10.0.0.0/16, east=10.1.0.0/16), 상대 클러스터 CIDR로 가는 경로가 없다.

두 클러스터가 동일한 Cilium 루트 CA를 공유

 keast get secret -n kube-system cilium-ca
keast delete secret -n kube-system cilium-ca
NAME        TYPE     DATA   AGE
cilium-ca   Opaque   2      46m
secret "cilium-ca" deleted


 loopy  ~/cilium-lab
 kubectl --context kind-west get secret -n kube-system cilium-ca -o yaml | \
kubectl --context kind-east create -f -
secret/cilium-ca created

 keast get secret -n kube-system cilium-ca
NAME        TYPE     DATA   AGE
cilium-ca   Opaque   2      3m34s

클러스터 메쉬

cilium clustermesh enable --service-type NodePort --enable-kvstoremesh=false --context kind-west
cilium clustermesh enable --service-type NodePort --enable-kvstoremesh=false --context kind-east


# 모니터링
cilium clustermesh status --context kind-west --wait  
cilium clustermesh status --context kind-east --wait

kwest get svc,ep -n kube-system clustermesh-apiserver --context kind-west
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                            TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/clustermesh-apiserver   NodePort   10.2.160.3   <none>        2379:32379/TCP   111s

NAME                              ENDPOINTS         AGE
endpoints/clustermesh-apiserver   10.0.1.127:2379   111s


 kwest get pod -n kube-system -owide | grep clustermesh
clustermesh-apiserver-5cf45db9cc-4lpzj       2/2     Running     0          2m13s   10.0.1.127     west-worker          <none>           <none>
clustermesh-apiserver-generate-certs-mtrrt   0/1     Completed   0          2m13s   192.168.97.2   west-worker          <none>           <none>

클러스터 연결

cilium clustermesh connect --context kind-west --destination-context kind-east


# 모니터링
cilium clustermesh status --context kind-west --wait
cilium clustermesh status --context kind-east --wait


# 확인
kubectl exec -it -n kube-system ds/cilium -c cilium-agent --context kind-west -- cilium-dbg troubleshoot clustermesh
kubectl exec -it -n kube-system ds/cilium -c cilium-agent --context kind-east -- cilium-dbg troubleshoot clustermesh

 helm get values -n kube-system cilium --kube-context kind-east
USER-SUPPLIED VALUES:
autoDirectNodeRoutes: true
bpf:
  masquerade: true
cluster:
  id: 2
  name: east
clustermesh:
  apiserver:
    kvstoremesh:
      enabled: false
    service:
      type: NodePort
    tls:
      auto:
        enabled: true
        method: cronJob
        schedule: 0 0 1 */4 *
  config:
    clusters:
    - ips:
      - 192.168.97.3
      name: west # 설정 추가됨
      port: 32379
    enabled: true
  useAPIServer: true
debug:
  enabled: true
endpointHealthChecking:
  enabled: false
healthChecking: false
ipMasqAgent:
  config:
    nonMasqueradeCIDRs:
    - 10.0.0.0/16
  enabled: true
ipam:
  mode: kubernetes
ipv4NativeRoutingCIDR: 10.1.0.0/16
k8sServiceHost: 192.168.97.4
k8sServicePort: 6443
kubeProxyReplacement: true
operator:
  replicas: 1
routingMode: native

 helm get values -n kube-system cilium --kube-context kind-west
USER-SUPPLIED VALUES:
autoDirectNodeRoutes: true
bpf:
  masquerade: true
cluster:
  id: 1
  name: west
clustermesh:
  apiserver:
    kvstoremesh:
      enabled: false
    service:
      type: NodePort
    tls:
      auto:
        enabled: true
        method: cronJob
        schedule: 0 0 1 */4 *
  config:
    clusters:
    - ips:
      - 192.168.97.4
      name: east # 설정 추가됨
      port: 32379
    enabled: true
  useAPIServer: true
debug:
  enabled: true
endpointHealthChecking:
  enabled: false
healthChecking: false
ipMasqAgent:
  config:
    nonMasqueradeCIDRs:
    - 10.1.0.0/16
  enabled: true
ipam:
  mode: kubernetes
ipv4NativeRoutingCIDR: 10.0.0.0/16
k8sServiceHost: 192.168.97.3
k8sServicePort: 6443
kubeProxyReplacement: true
operator:
  replicas: 1
routingMode: native

추가된 라우팅 확인

docker exec -it west-control-plane ip -c route
default via 192.168.97.1 dev eth0
10.0.0.0/24 via 10.0.0.235 dev cilium_host proto kernel src 10.0.0.235
10.0.0.235 dev cilium_host proto kernel scope link
10.0.1.0/24 via 192.168.97.2 dev eth0 proto kernel
10.1.0.0/24 via 192.168.97.4 dev eth0 proto kernel # 추가
10.1.1.0/24 via 192.168.97.5 dev eth0 proto kernel # 추가
192.168.97.0/24 dev eth0 proto kernel scope link src 192.168.97.3


docker exec -it east-control-plane ip -c route
default via 192.168.97.1 dev eth0
10.0.0.0/24 via 192.168.97.3 dev eth0 proto kernel
10.0.1.0/24 via 10.0.1.76 dev cilium_host proto kernel src 10.0.1.76
10.0.1.76 dev cilium_host proto kernel scope link
10.1.0.0/24 via 192.168.97.4 dev eth0 proto kernel # 추가
10.1.1.0/24 via 192.168.97.5 dev eth0 proto kernel # 추가
192.168.97.0/24 dev eth0 proto kernel scope link src 192.168.97.2


docker exec -it west-worker ip -c route
default via 192.168.97.1 dev eth0
10.0.0.0/24 via 192.168.97.3 dev eth0 proto kernel # 추가
10.0.1.0/24 via 192.168.97.2 dev eth0 proto kernel # 추가
10.1.0.0/24 via 10.1.0.7 dev cilium_host proto kernel src 10.1.0.7
10.1.0.7 dev cilium_host proto kernel scope link
10.1.1.0/24 via 192.168.97.5 dev eth0 proto kernel
192.168.97.0/24 dev eth0 proto kernel scope link src 192.168.97.4


docker exec -it east-worker ip -c route
default via 192.168.97.1 dev eth0
10.0.0.0/24 via 192.168.97.3 dev eth0 proto kernel # 추가
10.0.1.0/24 via 192.168.97.2 dev eth0 proto kernel # 추가
10.1.0.0/24 via 192.168.97.4 dev eth0 proto kernel
10.1.1.0/24 via 10.1.1.73 dev cilium_host proto kernel src 10.1.1.73
10.1.1.73 dev cilium_host proto kernel scope link
192.168.97.0/24 dev eth0 proto kernel scope link src 192.168.97.5

서로의 PodCIDR 경로(10.0/16 ↔ 10.1/16) 가 노드 라우팅 테이블에 추가되어 클러스터 간의 통신이 가능하게 되었다.

허블 설정

 helm upgrade cilium cilium/cilium --version 1.17.6 --namespace kube-system --reuse-values \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=30001 --kube-context kind-west
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Sat Aug 16 23:45:09 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.

Your release version is 1.17.6.

For any further help, visit https://docs.cilium.io/en/v1.17/gettinghelp


 kwest -n kube-system rollout restart ds/cilium
daemonset.apps/cilium restarted

 helm upgrade cilium cilium/cilium --version 1.17.6 --namespace kube-system --reuse-values \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=31001 --kube-context kind-east
kwest -n kube-system rollout restart ds/cilium
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Sat Aug 16 23:46:02 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.

Your release version is 1.17.6.

For any further help, visit https://docs.cilium.io/en/v1.17/gettinghelp
daemonset.apps/cilium restarted

파드 통신 테스트

cat << EOF | kubectl apply --context kind-west -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF

cat << EOF | kubectl apply --context kind-east -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF

 kwest get pod -owide && keast get pod -owide
NAME       READY   STATUS    RESTARTS   AGE   IP          NODE          NOMINATED NODE   READINESS GATES
curl-pod   1/1     Running   0          39s   10.0.1.14   west-worker   <none>           <none>
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
curl-pod   1/1     Running   0          39s   10.1.1.216   east-worker   <none>           <none>

 kubectl exec -it curl-pod --context kind-west -- ping -c 1 10.1.1.216
PING 10.1.1.216 (10.1.1.216) 56(84) bytes of data.
64 bytes from 10.1.1.216: icmp_seq=1 ttl=62 time=0.637 ms

--- 10.1.1.216 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.637/0.637/0.637/0.000 ms


 kubectl exec -it curl-pod --context kind-east -- ping -c 1 10.0.1.14
PING 10.0.1.14 (10.0.1.14) 56(84) bytes of data.
64 bytes from 10.0.1.14: icmp_seq=1 ttl=62 time=0.845 ms

--- 10.0.1.14 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.845/0.845/0.845/0.000 ms

로드밸런서 배포

cat << EOF | kubectl apply --context kind-west -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
  annotations:
    service.cilium.io/global: "true"
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF


cat << EOF | kubectl apply --context kind-east -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
  annotations:
    service.cilium.io/global: "true"
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF

 kwest get svc,ep webpod && keast get svc,ep webpod
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/webpod   ClusterIP   10.2.238.70   <none>        80/TCP    49s

NAME               ENDPOINTS                   AGE
endpoints/webpod   10.0.1.37:80,10.0.1.71:80   49s
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/webpod   ClusterIP   10.3.226.229   <none>        80/TCP    43s

NAME               ENDPOINTS                   AGE
endpoints/webpod   10.1.1.31:80,10.1.1.38:80   43s

 kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend                 Service Type   Backend
1    10.2.0.1:443/TCP         ClusterIP      1 => 192.168.97.3:6443/TCP (active)
2    10.2.118.236:443/TCP     ClusterIP      1 => 192.168.97.2:4244/TCP (active)
3    10.2.0.10:53/UDP         ClusterIP      1 => 10.0.1.157:53/UDP (active)
                                             2 => 10.0.1.233:53/UDP (active)
4    10.2.0.10:53/TCP         ClusterIP      1 => 10.0.1.157:53/TCP (active)
                                             2 => 10.0.1.233:53/TCP (active)
5    10.2.0.10:9153/TCP       ClusterIP      1 => 10.0.1.157:9153/TCP (active)
                                             2 => 10.0.1.233:9153/TCP (active)
6    10.2.160.3:2379/TCP      ClusterIP      1 => 10.0.1.127:2379/TCP (active)
7    192.168.97.2:32379/TCP   NodePort       1 => 10.0.1.127:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.0.1.127:2379/TCP (active)
9    10.2.232.181:80/TCP      ClusterIP      1 => 10.0.1.142:4245/TCP (active)
10   10.2.156.98:80/TCP       ClusterIP      1 => 10.0.1.254:8081/TCP (active)
11   192.168.97.2:30001/TCP   NodePort       1 => 10.0.1.254:8081/TCP (active)
12   0.0.0.0:30001/TCP        NodePort       1 => 10.0.1.254:8081/TCP (active)
###
13   10.2.238.70:80/TCP       ClusterIP      1 => 10.0.1.37:80/TCP (active)
                                             2 => 10.0.1.71:80/TCP (active)
                                             3 => 10.1.1.31:80/TCP (active)
                                             4 => 10.1.1.38:80/TCP (active)
 loopy  ~/cilium-lab
 keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend                 Service Type   Backend
1    10.3.0.1:443/TCP         ClusterIP      1 => 192.168.97.4:6443/TCP (active)
2    10.3.35.16:443/TCP       ClusterIP      1 => 192.168.97.5:4244/TCP (active)
3    10.3.0.10:53/TCP         ClusterIP      1 => 10.1.0.53:53/TCP (active)
                                             2 => 10.1.0.132:53/TCP (active)
4    10.3.0.10:9153/TCP       ClusterIP      1 => 10.1.0.53:9153/TCP (active)
                                             2 => 10.1.0.132:9153/TCP (active)
5    10.3.0.10:53/UDP         ClusterIP      1 => 10.1.0.53:53/UDP (active)
                                             2 => 10.1.0.132:53/UDP (active)
6    10.3.105.94:2379/TCP     ClusterIP      1 => 10.1.1.130:2379/TCP (active)
7    192.168.97.5:32379/TCP   NodePort       1 => 10.1.1.130:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.1.1.130:2379/TCP (active)
9    10.3.152.190:80/TCP      ClusterIP      1 => 10.1.1.147:4245/TCP (active)
10   10.3.76.135:80/TCP       ClusterIP      1 => 10.1.1.1:8081/TCP (active)
11   192.168.97.5:31001/TCP   NodePort       1 => 10.1.1.1:8081/TCP (active)
12   0.0.0.0:31001/TCP        NodePort       1 => 10.1.1.1:8081/TCP (active)
###
13   10.3.226.229:80/TCP      ClusterIP      1 => 10.0.1.37:80/TCP (active)
                                             2 => 10.0.1.71:80/TCP (active)
                                             3 => 10.1.1.31:80/TCP (active)
                                             4 => 10.1.1.38:80/TCP (active)

한 ClusterIP 서비스의 백엔드가 두 클러스터에 걸쳐 존재함으로써 kube-proxy 없이 Cilium이 VIP → Pod 매핑을 관리하고

로컬, 원격 파드를 동시에 엔드포인트로 가진 로드밸런싱이 가능하게 된다.

그리고 레플리카를 0으로 줄여도 통신은 정상적으로 수행되게 되는데,

서비스 엔드포인트를 클러스터 간에 동기화하는 클러스터 메쉬 형태로 동작하기 때문에 로컬 파드가 0개라도 원격 클러스터에 파드가 있으면 서비스는 정상적으로 동작할 수 있게 된다.

로컬 우선 라우팅(Local Affinity) 설정

 kwest annotate service webpod service.cilium.io/affinity=local --overwrite
service/webpod annotated
 loopy  ~/cilium-lab

keast annotate service webpod service.cilium.io/affinity=local --overwrite
service/webpod annotated

 kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity


ID   Frontend                 Service Type   Backend
1    10.2.0.1:443/TCP         ClusterIP      1 => 192.168.97.3:6443/TCP (active)
2    10.2.118.236:443/TCP     ClusterIP      1 => 192.168.97.2:4244/TCP (active)
3    10.2.0.10:53/UDP         ClusterIP      1 => 10.0.1.157:53/UDP (active)
                                             2 => 10.0.1.233:53/UDP (active)
4    10.2.0.10:53/TCP         ClusterIP      1 => 10.0.1.157:53/TCP (active)
                                             2 => 10.0.1.233:53/TCP (active)
5    10.2.0.10:9153/TCP       ClusterIP      1 => 10.0.1.157:9153/TCP (active)
                                             2 => 10.0.1.233:9153/TCP (active)
6    10.2.160.3:2379/TCP      ClusterIP      1 => 10.0.1.127:2379/TCP (active)
7    192.168.97.2:32379/TCP   NodePort       1 => 10.0.1.127:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.0.1.127:2379/TCP (active)
9    10.2.232.181:80/TCP      ClusterIP      1 => 10.0.1.142:4245/TCP (active)
10   10.2.156.98:80/TCP       ClusterIP      1 => 10.0.1.254:8081/TCP (active)
11   192.168.97.2:30001/TCP   NodePort       1 => 10.0.1.254:8081/TCP (active)
12   0.0.0.0:30001/TCP        NodePort       1 => 10.0.1.254:8081/TCP (active)
13   10.2.238.70:80/TCP       ClusterIP      1 => 10.1.1.31:80/TCP (active)
                                             2 => 10.1.1.38:80/TCP (active)
                                             3 => 10.0.1.239:80/TCP (active) (preferred)
                                             4 => 10.0.1.146:80/TCP (active) (preferred)
                                             
    
   keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity


ID   Frontend                 Service Type   Backend
1    10.3.0.1:443/TCP         ClusterIP      1 => 192.168.97.4:6443/TCP (active)
2    10.3.35.16:443/TCP       ClusterIP      1 => 192.168.97.5:4244/TCP (active)
3    10.3.0.10:53/TCP         ClusterIP      1 => 10.1.0.53:53/TCP (active)
                                             2 => 10.1.0.132:53/TCP (active)
4    10.3.0.10:9153/TCP       ClusterIP      1 => 10.1.0.53:9153/TCP (active)
                                             2 => 10.1.0.132:9153/TCP (active)
5    10.3.0.10:53/UDP         ClusterIP      1 => 10.1.0.53:53/UDP (active)
                                             2 => 10.1.0.132:53/UDP (active)
6    10.3.105.94:2379/TCP     ClusterIP      1 => 10.1.1.130:2379/TCP (active)
7    192.168.97.5:32379/TCP   NodePort       1 => 10.1.1.130:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.1.1.130:2379/TCP (active)
9    10.3.152.190:80/TCP      ClusterIP      1 => 10.1.1.147:4245/TCP (active)
10   10.3.76.135:80/TCP       ClusterIP      1 => 10.1.1.1:8081/TCP (active)
11   192.168.97.5:31001/TCP   NodePort       1 => 10.1.1.1:8081/TCP (active)
12   0.0.0.0:31001/TCP        NodePort       1 => 10.1.1.1:8081/TCP (active)
13   10.3.226.229:80/TCP      ClusterIP      1 => 10.1.1.31:80/TCP (active) (preferred)
                                             2 => 10.1.1.38:80/TCP (active) (preferred)
                                             3 => 10.0.1.239:80/TCP (active)
                                             4 => 10.0.1.146:80/TCP (active)

service.cilium.io/affinity=local은 ClusterMesh 서비스가 로컬 파드를 우선적으로 사용하는 것을 보장하는 설정이다.

 kwest scale deployment webpod --replicas 0
deployment.apps/webpod scaled


 loopy  ~/cilium-lab
 kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend                 Service Type   Backend
1    10.2.0.1:443/TCP         ClusterIP      1 => 192.168.97.3:6443/TCP (active)
2    10.2.118.236:443/TCP     ClusterIP      1 => 192.168.97.2:4244/TCP (active)
3    10.2.0.10:53/UDP         ClusterIP      1 => 10.0.1.157:53/UDP (active)
                                             2 => 10.0.1.233:53/UDP (active)
4    10.2.0.10:53/TCP         ClusterIP      1 => 10.0.1.157:53/TCP (active)
                                             2 => 10.0.1.233:53/TCP (active)
5    10.2.0.10:9153/TCP       ClusterIP      1 => 10.0.1.157:9153/TCP (active)
                                             2 => 10.0.1.233:9153/TCP (active)
6    10.2.160.3:2379/TCP      ClusterIP      1 => 10.0.1.127:2379/TCP (active)
7    192.168.97.2:32379/TCP   NodePort       1 => 10.0.1.127:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.0.1.127:2379/TCP (active)
9    10.2.232.181:80/TCP      ClusterIP      1 => 10.0.1.142:4245/TCP (active)
10   10.2.156.98:80/TCP       ClusterIP      1 => 10.0.1.254:8081/TCP (active)
11   192.168.97.2:30001/TCP   NodePort       1 => 10.0.1.254:8081/TCP (active)
12   0.0.0.0:30001/TCP        NodePort       1 => 10.0.1.254:8081/TCP (active)
13   10.2.238.70:80/TCP       ClusterIP      1 => 10.1.1.31:80/TCP (active)
                                             2 => 10.1.1.38:80/TCP (active)

replicas=0으로 줄여 로컬 파드가 사라지면 Cilium은 자동으로 원격 파드 엔드포인트만 남겨 서비스 트래픽을 처리한다.

이 덕분에 서비스는 다운되지 않고 클러스터 간 자동 failover가 동작한다.

원격 우선 설정

 kwest annotate service webpod service.cilium.io/affinity=remote --overwrite
keast annotate service webpod service.cilium.io/affinity=remote --overwrite
service/webpod annotated
service/webpod annotated


 loopy  ~/cilium-lab
 kwest describe svc webpod | grep Annotations -A3
keast describe svc webpod | grep Annotations -A3
Annotations:              service.cilium.io/affinity: remote
                          service.cilium.io/global: true
Selector:                 app=webpod
Type:                     ClusterIP
Annotations:              service.cilium.io/affinity: remote
                          service.cilium.io/global: true
Selector:                 app=webpod
Type:                     ClusterIP

 kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend                 Service Type   Backend
1    10.2.0.1:443/TCP         ClusterIP      1 => 192.168.97.3:6443/TCP (active)
2    10.2.118.236:443/TCP     ClusterIP      1 => 192.168.97.2:4244/TCP (active)
3    10.2.0.10:53/UDP         ClusterIP      1 => 10.0.1.157:53/UDP (active)
                                             2 => 10.0.1.233:53/UDP (active)
4    10.2.0.10:53/TCP         ClusterIP      1 => 10.0.1.157:53/TCP (active)
                                             2 => 10.0.1.233:53/TCP (active)
5    10.2.0.10:9153/TCP       ClusterIP      1 => 10.0.1.157:9153/TCP (active)
                                             2 => 10.0.1.233:9153/TCP (active)
6    10.2.160.3:2379/TCP      ClusterIP      1 => 10.0.1.127:2379/TCP (active)
7    192.168.97.2:32379/TCP   NodePort       1 => 10.0.1.127:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.0.1.127:2379/TCP (active)
9    10.2.232.181:80/TCP      ClusterIP      1 => 10.0.1.142:4245/TCP (active)
10   10.2.156.98:80/TCP       ClusterIP      1 => 10.0.1.254:8081/TCP (active)
11   192.168.97.2:30001/TCP   NodePort       1 => 10.0.1.254:8081/TCP (active)
12   0.0.0.0:30001/TCP        NodePort       1 => 10.0.1.254:8081/TCP (active)
13   10.2.238.70:80/TCP       ClusterIP      1 => 10.1.1.31:80/TCP (active) (preferred)
                                             2 => 10.1.1.38:80/TCP (active) (preferred)
                                             3 => 10.0.1.253:80/TCP (active)
                                             4 => 10.0.1.106:80/TCP (active)


keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend                 Service Type   Backend
1    10.3.0.1:443/TCP         ClusterIP      1 => 192.168.97.4:6443/TCP (active)
2    10.3.35.16:443/TCP       ClusterIP      1 => 192.168.97.5:4244/TCP (active)
3    10.3.0.10:53/TCP         ClusterIP      1 => 10.1.0.53:53/TCP (active)
                                             2 => 10.1.0.132:53/TCP (active)
4    10.3.0.10:9153/TCP       ClusterIP      1 => 10.1.0.53:9153/TCP (active)
                                             2 => 10.1.0.132:9153/TCP (active)
5    10.3.0.10:53/UDP         ClusterIP      1 => 10.1.0.53:53/UDP (active)
                                             2 => 10.1.0.132:53/UDP (active)
6    10.3.105.94:2379/TCP     ClusterIP      1 => 10.1.1.130:2379/TCP (active)
7    192.168.97.5:32379/TCP   NodePort       1 => 10.1.1.130:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.1.1.130:2379/TCP (active)
9    10.3.152.190:80/TCP      ClusterIP      1 => 10.1.1.147:4245/TCP (active)
10   10.3.76.135:80/TCP       ClusterIP      1 => 10.1.1.1:8081/TCP (active)
11   192.168.97.5:31001/TCP   NodePort       1 => 10.1.1.1:8081/TCP (active)
12   0.0.0.0:31001/TCP        NodePort       1 => 10.1.1.1:8081/TCP (active)
13   10.3.226.229:80/TCP      ClusterIP      1 => 10.1.1.31:80/TCP (active)
                                             2 => 10.1.1.38:80/TCP (active)
                                             3 => 10.0.1.253:80/TCP (active) (preferred)
                                             4 => 10.0.1.106:80/TCP (active) (preferred)

반대 클러스터의 파드를 우선 사용하도록 설정하는 옵션이다.

동일한 이름의 글로벌 서비스가 여러 네임스페이스에서 공유

 kwest annotate service webpod service.cilium.io/affinity=local --overwrite
keast annotate service webpod service.cilium.io/affinity=local --overwrite
service/webpod annotated
service/webpod annotated
 loopy  ~/cilium-lab
 kwest describe svc webpod | grep Annotations -A3
keast describe svc webpod | grep Annotations -A3
Annotations:              service.cilium.io/affinity: local
                          service.cilium.io/global: true
Selector:                 app=webpod
Type:                     ClusterIP
Annotations:              service.cilium.io/affinity: local
                          service.cilium.io/global: true
Selector:                 app=webpod
Type:                     ClusterIP

 kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend                 Service Type   Backend
1    10.2.0.1:443/TCP         ClusterIP      1 => 192.168.97.3:6443/TCP (active)
2    10.2.118.236:443/TCP     ClusterIP      1 => 192.168.97.2:4244/TCP (active)
3    10.2.0.10:53/UDP         ClusterIP      1 => 10.0.1.157:53/UDP (active)
                                             2 => 10.0.1.233:53/UDP (active)
4    10.2.0.10:53/TCP         ClusterIP      1 => 10.0.1.157:53/TCP (active)
                                             2 => 10.0.1.233:53/TCP (active)
5    10.2.0.10:9153/TCP       ClusterIP      1 => 10.0.1.157:9153/TCP (active)
                                             2 => 10.0.1.233:9153/TCP (active)
6    10.2.160.3:2379/TCP      ClusterIP      1 => 10.0.1.127:2379/TCP (active)
7    192.168.97.2:32379/TCP   NodePort       1 => 10.0.1.127:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.0.1.127:2379/TCP (active)
9    10.2.232.181:80/TCP      ClusterIP      1 => 10.0.1.142:4245/TCP (active)
10   10.2.156.98:80/TCP       ClusterIP      1 => 10.0.1.254:8081/TCP (active)
11   192.168.97.2:30001/TCP   NodePort       1 => 10.0.1.254:8081/TCP (active)
12   0.0.0.0:30001/TCP        NodePort       1 => 10.0.1.254:8081/TCP (active)
13   10.2.238.70:80/TCP       ClusterIP      1 => 10.1.1.31:80/TCP (active)
                                             2 => 10.1.1.38:80/TCP (active)
                                             3 => 10.0.1.253:80/TCP (active) (preferred)
                                             4 => 10.0.1.106:80/TCP (active) (preferred)


keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium service list --clustermesh-affinity
ID   Frontend                 Service Type   Backend
1    10.3.0.1:443/TCP         ClusterIP      1 => 192.168.97.4:6443/TCP (active)
2    10.3.35.16:443/TCP       ClusterIP      1 => 192.168.97.5:4244/TCP (active)
3    10.3.0.10:53/TCP         ClusterIP      1 => 10.1.0.53:53/TCP (active)
                                             2 => 10.1.0.132:53/TCP (active)
4    10.3.0.10:9153/TCP       ClusterIP      1 => 10.1.0.53:9153/TCP (active)
                                             2 => 10.1.0.132:9153/TCP (active)
5    10.3.0.10:53/UDP         ClusterIP      1 => 10.1.0.53:53/UDP (active)
                                             2 => 10.1.0.132:53/UDP (active)
6    10.3.105.94:2379/TCP     ClusterIP      1 => 10.1.1.130:2379/TCP (active)
7    192.168.97.5:32379/TCP   NodePort       1 => 10.1.1.130:2379/TCP (active)
8    0.0.0.0:32379/TCP        NodePort       1 => 10.1.1.130:2379/TCP (active)
9    10.3.152.190:80/TCP      ClusterIP      1 => 10.1.1.147:4245/TCP (active)
10   10.3.76.135:80/TCP       ClusterIP      1 => 10.1.1.1:8081/TCP (active)
11   192.168.97.5:31001/TCP   NodePort       1 => 10.1.1.1:8081/TCP (active)
12   0.0.0.0:31001/TCP        NodePort       1 => 10.1.1.1:8081/TCP (active)
13   10.3.226.229:80/TCP      ClusterIP      1 => 10.1.1.31:80/TCP (active) (preferred)
                                             2 => 10.1.1.38:80/TCP (active) (preferred)
                                             3 => 10.0.1.253:80/TCP (active)
                                             4 => 10.0.1.106:80/TCP (active)

East에서 접근했을 때 East에 파드가 있으면 그쪽이 (preferred), West에서 접근했을 때 West에 파드가 있으면 그쪽이 (preferred)

만약 East에 파드가 없으면 (replicas=0) 자동으로 West쪽 파드로 트래픽을 라우팅한다.

clustermesh-apiserver 파드 정보 확인

 kwest exec -it -n kube-system ds/cilium -c cilium-agent -- cilium node list
Name                      IPv4 Address   Endpoint CIDR   IPv6 Address   Endpoint CIDR   Source
east/east-control-plane   192.168.97.4   10.1.0.0/24                                    clustermesh
east/east-worker          192.168.97.5   10.1.1.0/24                                    clustermesh
west/west-control-plane   192.168.97.3   10.0.0.0/24                                    custom-resource
west/west-worker          192.168.97.2   10.0.1.0/24                                    local
 
 
 loopy  ~/cilium-lab
 keast exec -it -n kube-system ds/cilium -c cilium-agent -- cilium node list
Name                      IPv4 Address   Endpoint CIDR   IPv6 Address   Endpoint CIDR   Source
east/east-control-plane   192.168.97.4   10.1.0.0/24                                    custom-resource
east/east-worker          192.168.97.5   10.1.1.0/24                                    local
west/west-control-plane   192.168.97.3   10.0.0.0/24                                    clustermesh
west/west-worker          192.168.97.2   10.0.1.0/24                                    clustermesh

 kubectl describe pod -n kube-system -l k8s-app=clustermesh-apiserver
Name:                 clustermesh-apiserver-5cf45db9cc-qx428
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      clustermesh-apiserver
Node:                 east-worker/192.168.97.5
Start Time:           Sat, 16 Aug 2025 23:30:05 +0900
Labels:               app.kubernetes.io/name=clustermesh-apiserver
                      app.kubernetes.io/part-of=cilium
                      k8s-app=clustermesh-apiserver
                      pod-template-hash=5cf45db9cc
Annotations:          <none>
Status:               Running
IP:                   10.1.1.130
IPs:
  IP:           10.1.1.130
Controlled By:  ReplicaSet/clustermesh-apiserver-5cf45db9cc
Init Containers:
  etcd-init:
    Container ID:  containerd://ddb88891cf41e7ad9c4a838a3ba15de7a3776b2a22e357ce505614cc99d0b004
    Image:         quay.io/cilium/clustermesh-apiserver:v1.17.6@sha256:f619e97432db427e1511bf91af3be8ded418c53a353a09629e04c5880659d1df
    Image ID:      quay.io/cilium/clustermesh-apiserver@sha256:f619e97432db427e1511bf91af3be8ded418c53a353a09629e04c5880659d1df
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/clustermesh-apiserver
    Args:
      etcdinit
      --debug
      --etcd-cluster-name=clustermesh-apiserver
      --etcd-initial-cluster-token=$(INITIAL_CLUSTER_TOKEN)
      --etcd-data-dir=/var/run/etcd
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 16 Aug 2025 23:30:24 +0900
      Finished:     Sat, 16 Aug 2025 23:30:25 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      CILIUM_CLUSTER_NAME:    <set to the key 'cluster-name' of config map 'cilium-config'>  Optional: false
      INITIAL_CLUSTER_TOKEN:   (v1:metadata.uid)
    Mounts:
      /var/run/etcd from etcd-data-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9r75c (ro)
Containers:
  etcd:
    Container ID:  containerd://1204eedca147df5e7a2a0b2285e73a22de0780fba5e2d69395af6702aa8947b5
    Image:         quay.io/cilium/clustermesh-apiserver:v1.17.6@sha256:f619e97432db427e1511bf91af3be8ded418c53a353a09629e04c5880659d1df
    Image ID:      quay.io/cilium/clustermesh-apiserver@sha256:f619e97432db427e1511bf91af3be8ded418c53a353a09629e04c5880659d1df
    Ports:         2379/TCP, 9963/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /usr/bin/etcd
    Args:
      --data-dir=/var/run/etcd
      --name=clustermesh-apiserver
      --client-cert-auth
      --trusted-ca-file=/var/lib/etcd-secrets/ca.crt
      --cert-file=/var/lib/etcd-secrets/tls.crt
      --key-file=/var/lib/etcd-secrets/tls.key
      --listen-client-urls=https://127.0.0.1:2379,https://[$(HOSTNAME_IP)]:2379
      --advertise-client-urls=https://[$(HOSTNAME_IP)]:2379
      --initial-cluster-token=$(INITIAL_CLUSTER_TOKEN)
      --auto-compaction-retention=1
      --listen-metrics-urls=http://[$(HOSTNAME_IP)]:9963
      --metrics=basic
    State:          Running
      Started:      Sat, 16 Aug 2025 23:30:26 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      ETCDCTL_API:            3
      HOSTNAME_IP:             (v1:status.podIP)
      INITIAL_CLUSTER_TOKEN:   (v1:metadata.uid)
    Mounts:
      /var/lib/etcd-secrets from etcd-server-secrets (ro)
      /var/run/etcd from etcd-data-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9r75c (ro)
  apiserver:
    Container ID:  containerd://42ecb21e07a52347ca92fa60185e017ebd9d70c1daf4cce12b81f9feca665e2c
    Image:         quay.io/cilium/clustermesh-apiserver:v1.17.6@sha256:f619e97432db427e1511bf91af3be8ded418c53a353a09629e04c5880659d1df
    Image ID:      quay.io/cilium/clustermesh-apiserver@sha256:f619e97432db427e1511bf91af3be8ded418c53a353a09629e04c5880659d1df
    Ports:         9880/TCP, 9962/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /usr/bin/clustermesh-apiserver
    Args:
      clustermesh
      --debug
      --cluster-name=$(CLUSTER_NAME)
      --cluster-id=$(CLUSTER_ID)
      --kvstore-opt=etcd.config=/var/lib/cilium/etcd-config.yaml
      --kvstore-opt=etcd.qps=20
      --kvstore-opt=etcd.bootstrapQps=10000
      --max-connected-clusters=255
      --health-port=9880
      --enable-external-workloads=false
      --prometheus-serve-addr=:9962
      --controller-group-metrics=all
    State:          Running
      Started:      Sat, 16 Aug 2025 23:30:27 +0900
    Ready:          True
    Restart Count:  0
    Readiness:      http-get http://:apiserv-health/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      CLUSTER_NAME:               <set to the key 'cluster-name' of config map 'cilium-config'>               Optional: false
      CLUSTER_ID:                 <set to the key 'cluster-id' of config map 'cilium-config'>                 Optional: true
      ENABLE_K8S_ENDPOINT_SLICE:  <set to the key 'enable-k8s-endpoint-slice' of config map 'cilium-config'>  Optional: true
    Mounts:
      /var/lib/cilium/etcd-secrets from etcd-admin-client (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9r75c (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  etcd-server-secrets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          clustermesh-apiserver-server-cert
    SecretOptionalName:  <nil>
  etcd-admin-client:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          clustermesh-apiserver-admin-cert
    SecretOptionalName:  <nil>
  etcd-data-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-9r75c:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

실습 환경 삭제

kind delete cluster --name west && kind delete cluster --name east && kind delete cluster --name center

'Infra > 쿠버네티스' 카테고리의 다른 글

[Cilium] #6주차 (2) Gateway API Support (0)	2025.08.24
[Cilium] #6주차 (1) cilium + ingress (1)	2025.08.23
[Cilium] #5주차 (2) Cilium Geneve encapsulation + DSR(Direct Server Return) (5)	2025.08.16
[Cilium] #5주차 (1) Cilium BGP + ECMP를 활용한 쿠버네티스 로드밸런서 트래픽 라우팅 실습 (9)	2025.08.15
[Cilium] #4주차 (3) 오버레이 네트워크 (Geneve 모드) (1)	2025.08.10