[Cilium] #5주차 (1) Cilium BGP + ECMP를 활용한 쿠버네티스 로드밸런서 트래픽 라우팅 실습

25년도 Cilium Study 1기 정리 글입니다.

실습 세팅

샘플 앱 배포

(⎈|HomeLab:N/A) root@k8s-ctr:~# cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
EOFype: ClusterIP0rPort: 80miernetes.io/hostname"
deployment.apps/webpod created
service/webpod created
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  nodeName: k8s-ctr
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF
pod/curl-pod created

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get deploy,svc,ep webpod -owide
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES           SELECTOR
deployment.apps/webpod   3/3     3            3           19s   webpod       traefik/whoami   app=webpod

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/webpod   ClusterIP   10.96.180.250   <none>        80/TCP    19s   app=webpod

NAME               ENDPOINTS                                         AGE
endpoints/webpod   172.20.0.205:80,172.20.1.135:80,172.20.1.157:80   19s

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get endpointslices -l app=webpod
NAME           ADDRESSTYPE   PORTS   ENDPOINTS                                AGE
webpod-8kmfd   IPv4          80      172.20.0.205,172.20.1.135,172.20.1.157   26s

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ciliumendpoints
NAME                      SECURITY IDENTITY   ENDPOINT STATE   IPV4           IPV6
curl-pod                  15614               ready            172.20.0.187
webpod-697b545f57-42lfb   31003               ready            172.20.1.135
webpod-697b545f57-8r5sl   31003               ready            172.20.1.157
webpod-697b545f57-fnz6w   31003               ready            172.20.0.205

라우터 설정 확인

root@router:~# ss -tnlp | grep -iE 'zebra|bgpd'
LISTEN 0      3          127.0.0.1:2601      0.0.0.0:*    users:(("zebra",pid=4467,fd=25))
LISTEN 0      3          127.0.0.1:2605      0.0.0.0:*    users:(("bgpd",pid=4472,fd=18))
LISTEN 0      4096         0.0.0.0:179       0.0.0.0:*    users:(("bgpd",pid=4472,fd=22))
LISTEN 0      4096            [::]:179          [::]:*    users:(("bgpd",pid=4472,fd=23))



root@router:~# ps -ef |grep frr
root        4454       1  0 21:50 ?        00:00:01 /usr/lib/frr/watchfrr -d -F traditional zebra bgpd staticd
frr         4467       1  0 21:50 ?        00:00:00 /usr/lib/frr/zebra -d -F traditional -A 127.0.0.1 -s 90000000
frr         4472       1  0 21:50 ?        00:00:00 /usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1
frr         4479       1  0 21:50 ?        00:00:00 /usr/lib/frr/staticd -d -F traditional -A 127.0.0.1
root        4795    4776  0 22:33 pts/1    00:00:00 grep --color=auto frr

현재 실행 중인 프로세스

zebra: 커널 라우팅 테이블과 FRR 라우팅 정보를 동기화
bgpd: BGP 피어와 세션을 맺고 경로를 교환
staticd: 설정된 정적 라우트 관리
watchfrr: 위 데몬들 죽으면 자동 재시작

root@router:~# vtysh -c 'show running'
Building configuration...

Current configuration:
!
frr version 8.4.4
frr defaults traditional
hostname router
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
router bgp 65000
 bgp router-id 192.168.10.200 # BGP에서 식별자로 쓰는 고유 IP
 no bgp ebgp-requires-policy # eBGP에서 route-map 정책 없이도 경로 광고 허용
 bgp graceful-restart # 세션이 잠시 끊겨도 기존 라우팅 정보 유지
 bgp bestpath as-path multipath-relax # AS 경로 길이가 달라도 multipath 로드밸런싱 허용
 !
 address-family ipv4 unicast
  network 10.10.1.0/24
  maximum-paths 4
 exit-address-family
exit
!
end

root@router:~# cat /etc/frr/frr.conf
# default to using syslog. /etc/rsyslog.d/45-frr.conf places the log in
# /var/log/frr/frr.log
#
# Note:
# FRR's configuration shell, vtysh, dynamically edits the live, in-memory
# configuration while FRR is running. When instructed, vtysh will persist the
# live configuration to this file, overwriting its contents. If you want to
# avoid this, you can edit this file manually before starting FRR, or instruct
# vtysh to write configuration to a different file.
log syslog informational
!
router bgp 65000
  bgp router-id 192.168.10.200
  bgp graceful-restart
  no bgp ebgp-requires-policy
  bgp bestpath as-path multipath-relax
  maximum-paths 4
  network 10.10.1.0/24

root@router:~# ip -c route
default via 192.168.163.2 dev eth0 proto dhcp src 192.168.163.168 metric 100
10.10.1.0/24 dev loop1 proto kernel scope link src 10.10.1.200
10.10.2.0/24 dev loop2 proto kernel scope link src 10.10.2.200
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.200
192.168.20.0/24 dev eth2 proto kernel scope link src 192.168.20.200
192.168.163.0/24 dev eth0 proto kernel scope link src 192.168.163.168 metric 100
192.168.163.2 dev eth0 proto dhcp scope link src 192.168.163.168 metric 100


root@router:~# vtysh -c 'show ip route'
Codes: K - kernel route, C - connected, S - static, R - RIP, # 커널에서 가져온 기본 경로 / 직접 연결됨 / 정적 / 리얼 Ip
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/100] via 192.168.163.2, eth0, src 192.168.163.168, 01:05:55
C>* 10.10.1.0/24 is directly connected, loop1, 01:05:55
C>* 10.10.2.0/24 is directly connected, loop2, 01:05:55
C>* 192.168.10.0/24 is directly connected, eth1, 01:05:55
C>* 192.168.20.0/24 is directly connected, eth2, 01:05:55
C>* 192.168.163.0/24 [0/100] is directly connected, eth0, 01:05:55
K>* 192.168.163.2/32 [0/100] is directly connected, eth0, 01:05:55

Cilium BGP 피어 만들기

root@router:~# cat << EOF >> /etc/frr/frr.conf
  neighbor CILIUM peer-group
  neighbor CILIUM remote-as external
  neighbor 192.168.10.100 peer-group CILIUM
  neighbor 192.168.10.101 peer-group CILIUM
  neighbor 192.168.20.100 peer-group CILIUM
EOF


root@router:~# cat /etc/frr/frr.conf
# default to using syslog. /etc/rsyslog.d/45-frr.conf places the log in
# /var/log/frr/frr.log
#
# Note:
# FRR's configuration shell, vtysh, dynamically edits the live, in-memory
# configuration while FRR is running. When instructed, vtysh will persist the
# live configuration to this file, overwriting its contents. If you want to
# avoid this, you can edit this file manually before starting FRR, or instruct
# vtysh to write configuration to a different file.
log syslog informational
!
router bgp 65000
  bgp router-id 192.168.10.200
  bgp graceful-restart
  no bgp ebgp-requires-policy
  bgp bestpath as-path multipath-relax
  maximum-paths 4
  network 10.10.1.0/24
  neighbor CILIUM peer-group
  neighbor CILIUM remote-as external
  neighbor 192.168.10.100 peer-group CILIUM
  neighbor 192.168.10.101 peer-group CILIUM
  neighbor 192.168.20.100 peer-group CILIUM

systemctl daemon-reexec && systemctl restart frr
systemctl status frr --no-pager --full

컨트롤 플레인 BGP 동작할 노드 설정

# 라벨이 달린 노드에서만 BGP 기능을 킴
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl label nodes k8s-ctr k8s-w0 k8s-w1 enable-bgp=true
node/k8s-ctr labeled
node/k8s-w0 labeled
node/k8s-w1 labeled


(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get node -l enable-bgp=true
NAME      STATUS     ROLES           AGE   VERSION
k8s-ctr   Ready      control-plane   77m   v1.33.2
k8s-w0    NotReady   <none>          70m   v1.33.2
k8s-w1    Ready      <none>          74m   v1.33.2

cat << EOF | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisements
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: "PodCIDR"
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  timers:
    holdTimeSeconds: 9
    keepAliveTimeSeconds: 3
  ebgpMultihop: 2
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 15
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "bgp"
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchLabels:
      "enable-bgp": "true"
  bgpInstances:
  - name: "instance-65001"
    localASN: 65001
    peers:
    - name: "tor-switch"
      peerASN: 65000
      peerAddress: 192.168.10.200  # router ip address
      peerConfigRef:
        name: "cilium-peer"
EOF

컨트롤 플레인에서 라벨 붙은 노드들에서 BGP 스피커를 켜고,

로컬 AS는 65001로, 라우터(AS65000, 192.168.10.200)와 피어링하고, 각 노드의 PodCIDR를 광고하는 CRD를 적용한다.

라우터에서

journalctl -u frr -f

컨트롤 플레인에 BGP 피어 연결이 성공했고, 초기 라우트 교환이 완료됐다

192.168.10.101 (Cilium BGP 노드1) → IPv4 Unicast
192.168.10.100 (Cilium BGP 노드2) → IPv4 Unicast
192.168.20.100 (Cilium BGP 노드3) → IPv4 Unicast

적용된 내용 확인

(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium bgp peers

Node      Local AS   Peer AS   Peer Address     Session State   Uptime   Family         Received   Advertised
k8s-ctr   65001      65000     192.168.10.200   established     15m0s    ipv4/unicast   4          2
k8s-w0    65001      65000     192.168.10.200   established     13m51s   ipv4/unicast   4          2
k8s-w1    65001      65000     192.168.10.200   established     15m8s    ipv4/unicast   4          2


(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium bgp routes available ipv4 unicast
Node      VRouter   Prefix          NextHop   Age      Attrs
k8s-ctr   65001     172.20.0.0/24   0.0.0.0   15m13s   [{Origin: i} {Nexthop: 0.0.0.0}]
k8s-w0    65001     172.20.2.0/24   0.0.0.0   13m58s   [{Origin: i} {Nexthop: 0.0.0.0}]
k8s-w1    65001     172.20.1.0/24   0.0.0.0   15m13s   [{Origin: i} {Nexthop: 0.0.0.0}]

3개의 노드(k8s-ctr, k8s-w0, k8s-w1)가 모두 라우터와 BGP 세션을 맺고 있으며 IPv4 경로를 서로 주고받고 있는 것을 확인할 수 있다.

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ciliumbgpadvertisements,ciliumbgppeerconfigs,ciliumbgpclusterconfigs
NAME                                                  AGE
ciliumbgpadvertisement.cilium.io/bgp-advertisements   19m # 무엇을 광고할지 정의

NAME                                        AGE
ciliumbgppeerconfig.cilium.io/cilium-peer   19m # BGP 피어(라우터)와의 세션 파라미터 정의

NAME                                          AGE
ciliumbgpclusterconfig.cilium.io/cilium-bgp   19m # 어떤 노드에서 BGP를 켜고 어떤 AS/피어 설정을 쓸지 결정하는 리소스

root@router:~# ip -c route | grep bgp
172.20.0.0/24 nhid 28 via 192.168.10.100 dev eth1 proto bgp metric 20
172.20.1.0/24 nhid 26 via 192.168.10.101 dev eth1 proto bgp metric 20
172.20.2.0/24 nhid 30 via 192.168.20.100 dev eth2 proto bgp metric 20

# bgp 피어 상태
root@router:~# vtysh -c 'show ip bgp summary'
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.10.200, local AS number 65000 vrf-id 0
BGP table version 4
RIB entries 7, using 1344 bytes of memory
Peers 3, using 2172 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.168.10.100  4      65001       467       470        0    0    0 00:23:12            1        4 N/A
192.168.10.101  4      65001       470       473        0    0    0 00:23:20            1        4 N/A
192.168.20.100  4      65001       445       447        0    0    0 00:22:04            1        4 N/A

Total number of neighbors 3


root@router:~# vtysh -c 'show ip bgp'
BGP table version is 4, local router ID is 192.168.10.200, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.10.1.0/24     0.0.0.0                  0         32768 i        # 라우터 로컬에 있는 네트워크 (BGP로도 광고중)
*> 172.20.0.0/24    192.168.10.100                         0 65001 i  # k8s-ctr 노드의 PodCIDR
*> 172.20.1.0/24    192.168.10.101                         0 65001 i  # k8s-w1 노드의 PodCIDR
*> 172.20.2.0/24    192.168.20.100                         0 65001 i  # k8s-w0 노드의 PodCIDR

Displayed  4 routes and 4 total paths

라우터(FRR, 192.168.10.200)와 Cilium 노드(192.168.10.100) 간 BGP 세션이 Established가 된 상태이며, 라우터가 피어에게 여러 개의 경로를 한 번에 UPDATE 메시지로 전송하게된다.

노드에 라우팅 설정 추가하는 방법

(⎈|HomeLab:N/A) root@k8s-ctr:~# ip route add 172.20.0.0/16 via 192.168.10.200
(⎈|HomeLab:N/A) root@k8s-ctr:~# sshpass -p 'vagrant' ssh vagrant@k8s-w1 sudo ip route add 172.20.0.0/16 via 192.168.10.200
(⎈|HomeLab:N/A) root@k8s-ctr:~# sshpass -p 'vagrant' ssh vagrant@k8s-w0 sudo ip route add 172.20.0.0/16 via 192.168.20.200

워커 노드 커널 라우팅 테이블에 172.20.0.0/16 경로를 명시적으로 추가한 후 모든 Pod CIDR로 가는 트래픽이 NIC(eth1)을 통해 라우터로 나가게 된다.

root@router:~# ip -c route | grep bgp
172.20.0.0/24 nhid 60 via 192.168.10.100 dev eth1 proto bgp metric 20
172.20.1.0/24 nhid 59 via 192.168.10.101 dev eth1 proto bgp metric 20
172.20.2.0/24 nhid 56 via 192.168.20.100 dev eth2 proto bgp metric 20

설정 주입 이후 정상적으로 컬을 모두 수행하는 것을 확인할 수 있다.

Disabling CRD Status Report

(⎈|HomeLab:N/A) root@k8s-ctr:~# helm upgrade cilium cilium/cilium --version 1.18.0 --namespace kube-system --reuse-values \
  --set bgpControlPlane.statusReport.enabled=false
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Fri Aug 15 00:40:19 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.

Your release version is 1.18.0.

For any further help, visit https://docs.cilium.io/en/v1.18/gettinghelp

bgpControlPlane.statusReport.enabled=false 설정이 적용되어 CiliumBGPNodeConfig의 status 필드가 더 이상 주기적으로 갱신되지 않게 된다.

서비스 ip로 로드밸런서 생성하기

서비스에 할당할 LoadBalancer IP 대역을 Cilium에 등록

# Cilium이 Service=LoadBalancer에 할당할 외부 IP 풀(172.16.1.0/24)을 생성한다.
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "cilium-pool"
spec:
  allowFirstLastIPs: "No"
  blocks:
  - cidr: "172.16.1.0/24"
EOF

ciliumloadbalancerippool.cilium.io/cilium-pool created


(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ippool
NAME          DISABLED   CONFLICTING   IPS AVAILABLE   AGE
cilium-pool   false      False         254             5s

웹 파드 서비스타입을 로드밸런서로 변경

# 서비스를 로드밸런서로 전환한다
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl patch svc webpod -p '{"spec": {"type": "LoadBalancer"}}'
service/webpod patched


(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc webpod
NAME     TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
webpod   LoadBalancer   10.96.180.250   172.16.1.1    80:30921/TCP   26h


(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ippool
NAME          DISABLED   CONFLICTING   IPS AVAILABLE   AGE
cilium-pool   false      False         253             22s


(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe svc webpod | grep 'Traffic Policy'
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster


(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg service list
ID   Frontend                Service Type   Backend
1    10.96.0.1:443/TCP       ClusterIP      1 => 192.168.10.100:6443/TCP (active)
2    10.96.77.222:443/TCP    ClusterIP      1 => 192.168.20.100:4244/TCP (active)
3    10.96.170.16:80/TCP     ClusterIP      1 => 172.20.0.103:4245/TCP (active)
4    0.0.0.0:30003/TCP       NodePort       1 => 172.20.0.191:8081/TCP (active)
7    10.96.170.247:80/TCP    ClusterIP      1 => 172.20.0.191:8081/TCP (active)
8    10.96.0.10:53/TCP       ClusterIP      1 => 172.20.0.30:53/TCP (active)
                                            2 => 172.20.0.84:53/TCP (active)
9    10.96.0.10:53/UDP       ClusterIP      1 => 172.20.0.30:53/UDP (active)
                                            2 => 172.20.0.84:53/UDP (active)
10   10.96.0.10:9153/TCP     ClusterIP      1 => 172.20.0.30:9153/TCP (active)
                                            2 => 172.20.0.84:9153/TCP (active)
11   10.96.176.103:443/TCP   ClusterIP      1 => 172.20.0.128:10250/TCP (active)
12   10.96.180.250:80/TCP    ClusterIP      1 => 172.20.0.205:80/TCP (active)
                                            2 => 172.20.1.135:80/TCP (active)
                                            3 => 172.20.1.157:80/TCP (active)
20   0.0.0.0:30921/TCP       NodePort       1 => 172.20.0.205:80/TCP (active)
                                            2 => 172.20.1.135:80/TCP (active)
                                            3 => 172.20.1.157:80/TCP (active)
23   172.16.1.1:80/TCP       LoadBalancer   1 => 172.20.0.205:80/TCP (active)  # 로드밸런서 확인
                                            2 => 172.20.1.135:80/TCP (active)
                                            3 => 172.20.1.157:80/TCP (active)

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc webpod -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
LBIP=$(kubectl get svc webpod -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -s $LBIP
172.16.1.1Hostname: webpod-697b545f57-8r5sl
IP: 127.0.0.1
IP: ::1
IP: 172.20.1.157
IP: fe80::b042:54ff:fea3:7ccc
RemoteAddr: 192.168.10.100:40224
GET / HTTP/1.1
Host: 172.16.1.1
User-Agent: curl/8.5.0
Accept: */*

CiliumBGPAdvertisement 리소스 배포

BGP Advertisement 적용 전에는 라우터가 172.16.1.1로 가는 경로를 모르는 상태이다.

(⎈|HomeLab:N/A) root@k8s-ctr:~# cat << EOF | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisements-lb-exip-webpod
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: "Service"
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchExpressions:
          - { key: app, operator: In, values: [ webpod ] }
EOF
ciliumbgpadvertisement.cilium.io/bgp-advertisements-lb-exip-webpod created


(⎈|HomeLab:N/A) root@k8s-ctr:~#
kubectl get CiliumBGPAdvertisement
NAME                                AGE
bgp-advertisements                  25h
bgp-advertisements-lb-exip-webpod   4s

root@router:~# ip -c route
default via 192.168.163.2 dev eth0 proto dhcp src 192.168.163.168 metric 100
10.10.1.0/24 dev loop1 proto kernel scope link src 10.10.1.200
10.10.2.0/24 dev loop2 proto kernel scope link src 10.10.2.200
172.16.1.1 nhid 117 proto bgp metric 20                               # 추가
	nexthop via 192.168.20.100 dev eth2 weight 1                      # 추가
	nexthop via 192.168.10.101 dev eth1 weight 1                      # 추가
	nexthop via 192.168.10.100 dev eth1 weight 1                      # 추가
172.20.0.0/24 nhid 60 via 192.168.10.100 dev eth1 proto bgp metric 20
172.20.1.0/24 nhid 59 via 192.168.10.101 dev eth1 proto bgp metric 20
172.20.2.0/24 nhid 56 via 192.168.20.100 dev eth2 proto bgp metric 20
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.200
192.168.20.0/24 dev eth2 proto kernel scope link src 192.168.20.200
192.168.163.0/24 dev eth0 proto kernel scope link src 192.168.163.168 metric 100
192.168.163.2 dev eth0 proto dhcp scope link src 192.168.163.168 metric 100

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it -n kube-system ds/cilium -- cilium-dbg bgp route-policies
VRouter   Policy Name                                             Type     Match Peers         Match Families   Match Prefixes (Min..Max Len)   RIB Action   Path Actions
65001     allow-local                                             import                                                                        accept
65001     tor-switch-ipv4-PodCIDR                                 export   192.168.10.200/32                    172.20.2.0/24 (24..24)          accept
65001     tor-switch-ipv4-Service-webpod-default-LoadBalancerIP   export   192.168.10.200/32                    172.16.1.1/32 (32..32)          accept


(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium bgp routes available ipv4 unicast
Node      VRouter   Prefix          NextHop   Age      Attrs
k8s-ctr   65001     172.16.1.1/32   0.0.0.0   7m47s    [{Origin: i} {Nexthop: 0.0.0.0}]
          65001     172.20.0.0/24   0.0.0.0   23m20s   [{Origin: i} {Nexthop: 0.0.0.0}]
k8s-w0    65001     172.16.1.1/32   0.0.0.0   7m47s    [{Origin: i} {Nexthop: 0.0.0.0}]
          65001     172.20.2.0/24   0.0.0.0   23m20s   [{Origin: i} {Nexthop: 0.0.0.0}]
k8s-w1    65001     172.16.1.1/32   0.0.0.0   7m47s    [{Origin: i} {Nexthop: 0.0.0.0}]
          65001     172.20.1.0/24   0.0.0.0   23m7s    [{Origin: i} {Nexthop: 0.0.0.0}]

이 경로가 생김으로써 외부 라우터가 172.16.1.1로 트래픽을 보낼 수 있게 된 것이다.

root@router:~# sudo vtysh -c 'show ip bgp'
BGP table version is 8, local router ID is 192.168.10.200, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.10.1.0/24     0.0.0.0                  0         32768 i
*= 172.16.1.1/32    192.168.10.101                         0 65001 i   # 광고한 서비스 로드밸런서 Ip
*>                  192.168.10.100                         0 65001 i   # 메인 경로
*=                  192.168.20.100                         0 65001 i
*> 172.20.0.0/24    192.168.10.100                         0 65001 i
*> 172.20.1.0/24    192.168.10.101                         0 65001 i
*> 172.20.2.0/24    192.168.20.100                         0 65001 i

Displayed  5 routes and 7 total paths


root@router:~# sudo vtysh -c 'show ip bgp 172.16.1.1/32'
BGP routing table entry for 172.16.1.1/32, version 8
Paths: (3 available, best #2, table default)
  Advertised to non peer-group peers:
  192.168.10.100 192.168.10.101 192.168.20.100
  65001
    192.168.10.101 from 192.168.10.101 (192.168.10.101)
      Origin IGP, valid, external, multipath
      Last update: Fri Aug 15 00:56:05 2025
  65001
    192.168.10.100 from 192.168.10.100 (192.168.10.100)
      Origin IGP, valid, external, multipath, best (Router ID)
      Last update: Fri Aug 15 00:56:05 2025
  65001
    192.168.20.100 from 192.168.20.100 (192.168.20.100)
      Origin IGP, valid, external, multipath
      Last update: Fri Aug 15 00:56:05 2025

172.16.1.1/32가 외부 라우터의 BGP 테이블에 등록되어 외부 네트워크에서 이 IP로 접근 가능하다. 3개의 노드가 모두 next hop으로 광고되어 ECMP로 부하분산 동작이 가능하다.

Best Path는 192.168.10.100이지만, multipath 설정이 있으면 나머지도 동시에 사용 할 수 있다.

라우터에서 통신 확인

root@router:~# LBIP=172.16.1.1
curl -s $LBIP
Hostname: webpod-697b545f57-42lfb
IP: 127.0.0.1
IP: ::1
IP: 172.20.1.135
IP: fe80::f89a:ebff:fecd:683a
RemoteAddr: 192.168.20.100:53018
GET / HTTP/1.1
Host: 172.16.1.1
User-Agent: curl/8.5.0
Accept: */*

문제 확인

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl scale deployment webpod --replicas 2
deployment.apps/webpod scaled


(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pod -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP             NODE      NOMINATED NODE   READINESS GATES
curl-pod                  1/1     Running   0          26h   172.20.0.187   k8s-ctr   <none>           <none>
webpod-697b545f57-42lfb   1/1     Running   0          26h   172.20.1.135   k8s-w1    <none>           <none>
webpod-697b545f57-fnz6w   1/1     Running   0          26h   172.20.0.205   k8s-ctr   <none>           <none>


(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium bgp routes
(Defaulting to `available ipv4 unicast` routes, please see help for more options)

Node      VRouter   Prefix          NextHop   Age      Attrs
k8s-ctr   65001     172.16.1.1/32   0.0.0.0   15m0s    [{Origin: i} {Nexthop: 0.0.0.0}]
          65001     172.20.0.0/24   0.0.0.0   30m33s   [{Origin: i} {Nexthop: 0.0.0.0}]
k8s-w0    65001     172.16.1.1/32   0.0.0.0   15m0s    [{Origin: i} {Nexthop: 0.0.0.0}]
          65001     172.20.2.0/24   0.0.0.0   30m33s   [{Origin: i} {Nexthop: 0.0.0.0}]
k8s-w1    65001     172.16.1.1/32   0.0.0.0   15m0s    [{Origin: i} {Nexthop: 0.0.0.0}]
          65001     172.20.1.0/24   0.0.0.0   30m20s   [{Origin: i} {Nexthop: 0.0.0.0}]

Cilium BGP를 통해 LoadBalancer IP를 모든 노드가 광고하지만 실제로 서비스 파드가 없는 노드(w0)도 광고를 하고 있어서 불필요한 트래픽이 해당 노드로 유입될 수 있다.

External Traffic Policy (Local)

Cilium이 파드가 있는 노드만 BGP 광고하여 192.168.20.100(k8s-w0)은 광고 대상에서 제외된다.

따라서 라우터가 해당 노드로는 더 이상 트래픽을 보내지 않게 된다.

externalTrafficPolicy: Local은 서비스의 로드밸런서 IP를 광고할 때 실제 파드가 있는 노드만 타겟하여 BGP 라우팅에서 불필요한 경로를 제거하고 서비스 접근 경로를 최적화할 수 있다.

ECMP hash policy

root@router:~# sudo sysctl -w net.ipv4.fib_multipath_hash_policy=1
echo "net.ipv4.fib_multipath_hash_policy=1" >> /etc/sysctl.conf
net.ipv4.fib_multipath_hash_policy = 1


root@router:~# for i in {1..100};  do curl -s $LBIP | grep Hostname; done | sort | uniq -c | sort -nr
     36 Hostname: webpod-697b545f57-22gbp
     33 Hostname: webpod-697b545f57-42lfb
     31 Hostname: webpod-697b545f57-fnz6w

ECMP 라우팅은 동일한 목적지로 가는 여러 경로를 동시에 사용하여 부하를 분산하는데, 리눅스에서는 기본적으로 (소스IP, 목적지IP, L4포트)를 해시 키로 사용하여 다음 홉을 결정하지만 'fib_multipath_hash_policy=1'로 변경하면 포트를 무시하고 (소스IP, 목적지IP)만 사용한다.

이렇게 사용함으로써 포트 기반 해시 편향을 줄여 부하 분산을 더 균등하게 만들 수 있다.

'Infra > 쿠버네티스' 카테고리의 다른 글

[Cilium] #5주차 (3) 클러스터 메쉬 (1)	2025.08.17
[Cilium] #5주차 (2) Cilium Geneve encapsulation + DSR(Direct Server Return) (5)	2025.08.16
[Cilium] #4주차 (3) 오버레이 네트워크 (Geneve 모드) (1)	2025.08.10
[Cilium] #4주차 (2) Service LB-IPAM (8)	2025.08.10
[Cilium] #4주차 (1) 노드에 파드들간 통신 (네이티브 라우팅, 오버레이 네트워크 VXLAN) (4)	2025.08.07