[Cilium] #2주차 (4) Hubble Flow Logs를 Grafana로 시각화하기 (Loki + Promtail)

Hubble Exporter

hubble observe 명령은 CLI에서 실시간 흐름을 확인할 수 있지만, 장기 저장을 위해서는 Hubble Exporter를 사용해 Flow Log를 외부 시스템(파일, Kafka, Loki 등)으로 내보낼 수 있다.

Exporter는 hubble-relay와 cilium-agent 간 gRPC 스트림을 활용하여 Flow를 받아온다.

Cilium Hubble → Promtail → Loki → Grafana로 매트릭을 볼 수 있도록 구성해본다!

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get cm -n kube-system cilium-config -o json | grep hubble-export
        "hubble-export-allowlist": "",
        "hubble-export-denylist": "",
        "hubble-export-fieldmask": "",
        "hubble-export-file-max-backups": "5",
        "hubble-export-file-max-size-mb": "10",
        "hubble-export-file-path": "/var/run/cilium/hubble/events.log",

helm upgrade

helm upgrade cilium cilium/cilium \
  -n kube-system \
  --reuse-values \
  --set hubble.enabled=true \
  --set hubble.export.dynamic.enabled=true \
  --set hubble.export.dynamic.config.content[0].name=system \
  --set hubble.export.dynamic.config.content[0].filePath=/var/run/cilium/hubble/events-system.log \
  --set hubble.export.dynamic.config.content[0].includeFilters[0].source_pod[0]='kube_system/' \
  --set hubble.export.dynamic.config.content[0].includeFilters[1].destination_pod[0]='kube_system/'

hubble flow 저장 위치

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl -n kube-system get configmap cilium-config -o yaml | grep -A10 hubble-export
  hubble-export-allowlist: ""
  hubble-export-denylist: ""
  hubble-export-fieldmask: ""
  hubble-export-file-max-backups: "5"
  hubble-export-file-max-size-mb: "10"
  hubble-export-file-path: /var/run/cilium/hubble/events.log # 허블 플로우 저장 위치
  hubble-flowlogs-config-path: /flowlog-config/flowlogs.yaml
  hubble-listen-address: :4244
  hubble-metrics: dns drop tcp flow port-distribution icmp httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
  hubble-metrics-server: :9965
  hubble-metrics-server-enable-tls: "false"
  hubble-socket-path: /var/run/cilium/hubble.sock
  hubble-tls-cert-file: /var/lib/cilium/tls/hubble/server.crt
  hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/client-ca.crt
  hubble-tls-key-file: /var/lib/cilium/tls/hubble/server.key
  identity-allocation-mode: crd

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- tail -f /var/run/cilium/hubble/events.log

현재 저장되고 있는 이벤트를 확인할 수 있다.

Loki 설치

(⎈|HomeLab:N/A) root@k8s-ctr:~# helm upgrade --install loki grafana/loki-stack \
  --set grafana.enabled=true \
  -n monitoring --create-namespace
Release "loki" does not exist. Installing it now.
NAME: loki
LAST DEPLOYED: Sat Jul 26 21:52:29 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.

See http://docs.grafana.org/features/datasources/loki/ for more detail.

promtail 설치

(⎈|HomeLab:N/A) root@k8s-ctr:~# helm upgrade --install promtail grafana/promtail \
  -n monitoring \
  --create-namespace \
  --set "config.clients[0].url=http://loki.monitoring:3100/loki/api/v1/push" \
  --set "extraVolumes[0].name=hubble-logs" \
  --set "extraVolumes[0].hostPath.path=/var/run/cilium/hubble" \
  --set "extraVolumeMounts[0].name=hubble-logs" \
  --set "extraVolumeMounts[0].mountPath=/var/run/cilium/hubble" \
  --set "config.scrape_configs[1].job_name=hubble" \
  --set "config.scrape_configs[1].static_configs[0].targets[0]=localhost" \
  --set "config.scrape_configs[1].static_configs[0].labels.job=hubble" \
  --set "config.scrape_configs[1].static_configs[0].labels.__path__=/var/run/cilium/hubble/events*.log"
Release "promtail" does not exist. Installing it now.
NAME: promtail
LAST DEPLOYED: Sat Jul 26 23:30:38 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
***********************************************************************
 Welcome to Grafana Promtail
 Chart version: 6.17.0
 Promtail version: 3.5.1
***********************************************************************

Verify the application is working by running these commands:
* kubectl --namespace monitoring port-forward daemonset/promtail 3101
* curl http://127.0.0.1:3101/metrics

Promtail을 사용해 Hubble의 events.log 파일을 Loki에 전송한다.

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl -n monitoring describe pod -l app.kubernetes.io/name=promtail | grep -A5 "Mounts"
    Mounts:
      /etc/promtail from config (rw)
      /run/promtail from run (rw)
      /var/lib/docker/containers from containers (ro)
      /var/log/pods from pods (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jvx7l (ro)
--
    Mounts:
      /etc/promtail from config (rw)
      /run/promtail from run (rw)
      /var/lib/docker/containers from containers (ro)
      /var/log/pods from pods (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4t8m5 (ro)
--
    Mounts:
      /etc/promtail from config (rw)
      /run/promtail from run (rw)
      /var/lib/docker/containers from containers (ro)
      /var/log/pods from pods (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzgv5 (ro)
--
    Mounts:
      /etc/promtail from config (rw)
      /run/promtail from run (rw)
      /var/lib/docker/containers from containers (ro)
      /var/log/pods from pods (ro)
      /var/run/cilium/hubble from hubble-logs (rw)
--
    Mounts:
      /etc/promtail from config (rw)
      /run/promtail from run (rw)
      /var/lib/docker/containers from containers (ro)
      /var/log/pods from pods (ro)
      /var/run/cilium/hubble from hubble-logs (rw)
--
    Mounts:
      /etc/promtail from config (rw)
      /run/promtail from run (rw)
      /var/lib/docker/containers from containers (ro)
      /var/log/pods from pods (ro)
      /var/run/cilium/hubble from hubble-logs (rw)

Promtail이 Kubernetes 노드와 Pod 로그를 읽을 수 있도록 필요한 경로를 컨테이너 내부로 마운트된 설정을 확인할 수 있다.

helm get values promtail -n monitoring > promtail-values.yaml

(⎈|HomeLab:N/A) root@k8s-ctr:~# cat promtail-values.yaml
USER-SUPPLIED VALUES:
config:
  clients:
  - url: http://loki.monitoring:3100/loki/api/v1/push
  scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
    - role: pod
    pipeline_stages:
    - cri: {}
    relabel_configs:
    - action: replace
      regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
      source_labels:
      - __meta_kubernetes_pod_controller_name
      target_label: __tmp_controller_name
    - action: replace
      regex: ^;*([^;]+)(;.*)?$
      source_labels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_name
      - __meta_kubernetes_pod_label_app
      - __tmp_controller_name
      - __meta_kubernetes_pod_name
      target_label: app
    - action: replace
      regex: ^;*([^;]+)(;.*)?$
      source_labels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_instance
      - __meta_kubernetes_pod_label_instance
      target_label: instance
    - action: replace
      regex: ^;*([^;]+)(;.*)?$
      source_labels:
      - __meta_kubernetes_pod_label_app_kubernetes_io_component
      - __meta_kubernetes_pod_label_component
      target_label: component
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_node_name
      target_label: node_name
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: namespace
    - action: replace
      replacement: $1
      separator: /
      source_labels:
      - namespace
      - app
      target_label: job
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_name
      target_label: pod
    - action: replace
      source_labels:
      - __meta_kubernetes_pod_container_name
      target_label: container
    - action: replace
      replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
      - __meta_kubernetes_pod_uid
      - __meta_kubernetes_pod_container_name
      target_label: __path__
    - action: replace
      regex: true/(.*)
      replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
      - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
      - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
      - __meta_kubernetes_pod_container_name
      target_label: __path__
  - job_name: hubble
    static_configs:
    - labels:
        __path__: /var/run/cilium/hubble/events*.log
        job: hubble
      targets:
      - localhost
extraVolumeMounts:
- mountPath: /var/run/cilium/hubble
  name: hubble-logs
  readOnly: true
extraVolumes:
- hostPath:
    path: /var/run/cilium/hubble
    type: DirectoryOrCreate
  name: hubble-logs

프롬테일에 적용된 값이다.

근데 다소 궁금한점은.. 프롬테일 파드 내부 프롬테일 야믈 설정 파일 내에 내가 작성한 컨피그가 없다는 것... (/var/run/cilium/hubble/events*.log)

요건 내가 별도로 선언해서 파드 내부 컨피그에 덮어씌워지진 않고 관리되는 것일까..?

그라파나 노드 포트 설정

(⎈|HomeLab:N/A) root@k8s-ctr:~# k get po -A | grep grafana
monitoring    loki-grafana-664849db4c-gwf4r      2/2     Running   0          71s

(⎈|HomeLab:N/A) root@k8s-ctr:~kubectl patch svc loki-grafana -n monitoring -p '{"spec": {"type": "NodePort"}}'}'
kubectl get svc loki-grafana -n monitoring
service/loki-grafana patched
NAME           TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
loki-grafana   NodePort   10.96.103.176   <none>        80:30723/TCP   5m50s

노드 포트를 설정하여 로컬에서 접속확인해본다.

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get secret loki-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 --decode; echo

헬름으로 설치할 경우 쿠버네티스 시크릿에 저장돼있으므로 값을 추출한다. 위에 나온 값을 admin 사용자의 패스워드로 입력하여 로그인한다.

그라파나에서 Unable to connect with Loki 에러 발생할 때

Helm으로 loki-stack을 배포했지만 Grafana에서 Loki Data Source를 추가 시 그라파나에서 위와 같은 에러가 뜨고 그라파나 파드에 아래와 같이 로그가 남았다.

parse error at line 1, col 1: syntax error: unexpected IDENTIFIER

그런데 파드 내부에서 직접적으로 로키에는 통신이 되었다.

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it -n monitoring deploy/loki-grafana -c grafana -- sh
/usr/share/grafana $ curl http://loki.monitoring:3100/ready
ready
/usr/share/grafana $ curl http://loki.monitoring:3100/loki/api/v1/labels
{"status":"success","data":["app","component","container","filename","instance","job","namespace","node_name","pod","stream"]}

https://github.com/grafana/grafana/issues/79846

Grafana Loki Integration: Health check fails when Loki is up and functional · Issue #79846 · grafana/grafana

What happened? When attempting to add a new loki datasource via grafana, I get this error in the console, and a 'failed to connect' on the webUI. logger=tsdb.loki endpoint=checkHealth pluginId=loki...

github.com

비슷한 사례를 깃에서 찾아보니 있었는데 버전 종속성이 문제였다..!

loki-stack 차트가 설치한 Grafana 버전(10.x)이 Loki 2.9.x와 호환성이 좋지 않아 Health Check가 정상적으로 동작하지 않았던 것이다.

helm upgrade --install loki grafana/loki-stack \
  -n monitoring \
  --set grafana.enabled=true \
  --set grafana.image.tag=9.2.3
  
  
 (⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pods -n monitoring | grep grafana
loki-grafana-c9748bdd-wmxkk   2/2     Running   0          59s

그래서 로키와 호환되는 버전의 그라파나로 다운그레이드하여 다시 설치해주고,

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl patch svc loki-grafana -n monitoring -p '{"spec": {"type": "NodePort"}}'
service/loki-grafana patched

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl patch svc loki -n monitoring -p '{"spec": {"type": "NodePort"}}'
service/loki patched

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc -n monitoring
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
loki              NodePort    10.96.171.35   <none>        3100:30642/TCP   51m
loki-grafana      NodePort    10.96.107.66   <none>        80:32576/TCP     118s
loki-headless     ClusterIP   None           <none>        3100/TCP         51m
loki-memberlist   ClusterIP   None           <none>        7946/TCP         51m

노드포트도 다시 세팅한다.

제대로 연결이 되었다.

그라파나에서 쿼리하기

(⎈|HomeLab:N/A) root@k8s-ctr:~# curl "http://192.168.10.101:30642/loki/api/v1/labels"
{"status":"success","data":["app","component","container","filename","instance","job","namespace","node_name","pod","stream"]}

(⎈|HomeLab:N/A) root@k8s-ctr:~# curl "http://192.168.10.101:30642/loki/api/v1/label/job/values"
{"status":"success","data":["hubble","kube-system/cilium-agent","kube-system/cilium-operator","kube-system/coredns","kube-system/hubble-relay","kube-system/hubble-ui","kube-system/k8s-ctr","kube-system/promtail","monitoring/grafana","monitoring/loki","monitoring/promtail"]}

우선 로키에서 추출할 수 있는 매트릭을 먼저 컬해서 파악해본다.

{job="hubble"}로 쿼리하면 결과값이 잘 나오는 것을 볼 수 있다.

'Infra > 쿠버네티스' 카테고리의 다른 글

[Cilium] #3주차 (2) eBPF 기반 Cilium에서 SNAT 예외 설정 (ip-masq-agent) (3)	2025.08.01
[Cilium] #3주차 (1) Routing, Masquerading (3)	2025.07.30
[Cilium] #2주차 (3) 스타워즈 앱 배포 (L7 정책) (3)	2025.07.26
[Cilium] #2주차 (2) 스타워즈 앱 배포 (L3, L4 정책) (2)	2025.07.26
[Cilium] #2주차 (1) Hubble (2)	2025.07.26