[AEWS] #4주차 EKS 로깅 & AWS Cloud Watch 실습 (1)

25년도 AWS EKS Hands-on Study 스터디 정리 내용입니다.

 

EKS 로깅 

aws eks update-cluster-config --region ap-northeast-2 --name $CLUSTER_NAME \
    --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
    
aws logs describe-log-groups | jq

EKS의 로깅을 활성화하고 생성된 로그 그룹을 확인한다.

 

 

 

 

로그 인사이트

 

로그 인사이트를 통해 필터링을 해서 보면 kube-apiserver-audit 로그가 스캔된 것을 확인할 수 있다.

 

Nginx 컨테이너 배포

테스트를 위한 파드 배포

kubectl get ingress,deploy,svc,ep nginx
NAME                              CLASS   HOSTS                 ADDRESS                                                         PORTS   AGE
ingress.networking.k8s.io/nginx   alb     nginx.$MyDomain       myeks-ingress-alb-1318062263.ap-northeast-2.elb.amazonaws.com   80      2m31s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           2m31s

NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/nginx   NodePort   10.100.186.235   <none>        80:32219/TCP,443:31660/TCP   2m31s

NAME              ENDPOINTS                               AGE
endpoints/nginx   192.168.3.115:8443,192.168.3.115:8080   2m31s

kubectl describe deploy nginx
Name:                   nginx
Namespace:              default
CreationTimestamp:      Thu, 27 Feb 2025 22:10:33 +0900
Labels:                 app.kubernetes.io/instance=nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=nginx
                        app.kubernetes.io/version=1.27.4
                        helm.sh/chart=nginx-19.0.0
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: nginx
                        meta.helm.sh/release-namespace: default
Selector:               app.kubernetes.io/instance=nginx,app.kubernetes.io/name=nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=nginx
                    app.kubernetes.io/version=1.27.4
                    helm.sh/chart=nginx-19.0.0
  Service Account:  nginx
  Init Containers:
   preserve-logs-symlinks:
    Image:           docker.io/bitnami/nginx:1.27.4-debian-12-r1
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
    Args:
      -ec
      #!/bin/bash
      . /opt/bitnami/scripts/libfs.sh
      # We copy the logs folder because it has symlinks to stdout and stderr
      if ! is_dir_empty /opt/bitnami/nginx/logs; then
        cp -r /opt/bitnami/nginx/logs /emptydir/app-logs-dir
      fi

    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Environment:          <none>
    Mounts:
      /emptydir from empty-dir (rw)
  Containers:
   nginx:
    Image:           docker.io/bitnami/nginx:1.27.4-debian-12-r1
    Ports:           8080/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Liveness:             tcp-socket :http delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:            http-get http://:http/ delay=5s timeout=3s period=5s #success=1 #failure=3
    Environment:
      BITNAMI_DEBUG:            false
      NGINX_HTTP_PORT_NUMBER:   8080
      NGINX_HTTPS_PORT_NUMBER:  8443
    Mounts:
      /certs from certificate (rw)
      /opt/bitnami/nginx/conf from empty-dir (rw,path="app-conf-dir")
      /opt/bitnami/nginx/logs from empty-dir (rw,path="app-logs-dir")
      /opt/bitnami/nginx/tmp from empty-dir (rw,path="app-tmp-dir")
      /tmp from empty-dir (rw,path="tmp-dir")
  Volumes:
   empty-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   certificate:
    Type:          Secret (a volume populated by a Secret)
    SecretName:    nginx-tls
    Optional:      false
  Node-Selectors:  <none>
  Tolerations:     <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-7c94c9bdcb (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  2m39s  deployment-controller  Scaled up replica set nginx-7c94c9bdcb to 1

 

nginx 파드를 배포한 후 성공적으로 실행 중임을 확인할 수 있다.

여기서 이벤트를 보면 ScalingReplicaSet이라고 뜨는데 이는 ReplicaSet의 크기가 조정되었음을 의미한다.

 

curl -s https://nginx.$MyDomain
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

 

정상적으로 배포되었는지 확인한다.

 

kubectl stern deploy/nginx
+ nginx-7c94c9bdcb-44hzp › nginx
+ nginx-7c94c9bdcb-44hzp › preserve-logs-symlinks
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.31 INFO  ==>
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.31 INFO  ==> Welcome to the Bitnami nginx container
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.39 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.39 INFO  ==> Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami/ for more information.
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.40 INFO  ==>
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.40 INFO  ==> ** Starting NGINX setup **
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.49 INFO  ==> Validating settings in NGINX_* env vars
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.59 INFO  ==> No custom scripts in /docker-entrypoint-initdb.d
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.60 INFO  ==> Initializing NGINX
nginx-7c94c9bdcb-44hzp nginx realpath: /bitnami/nginx/conf/vhosts: No such file or directory
nginx-7c94c9bdcb-44hzp nginx
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.79 INFO  ==> ** NGINX setup finished! **
nginx-7c94c9bdcb-44hzp nginx nginx 13:10:43.81 INFO  ==> ** Starting NGINX **
nginx-7c94c9bdcb-44hzp nginx 192.168.3.32 - - [27/Feb/2025:13:10:49 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.31+" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.3.32 - - [27/Feb/2025:13:10:54 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.31+" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.3.32 - - [27/Feb/2025:13:10:59 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.31+" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.3.250 - - [27/Feb/2025:13:11:01 +0000] "GET / HTTP/1.1" 200 409 "-" "ELB-HealthChecker/2.0" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.1.4 - - [27/Feb/2025:13:11:01 +0000] "GET / HTTP/1.1" 200 409 "-" "ELB-HealthChecker/2.0" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.2.80 - - [27/Feb/2025:13:11:01 +0000] "GET / HTTP/1.1" 200 409 "-" "ELB-HealthChecker/2.0" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.3.32 - - [27/Feb/2025:13:11:04 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.31+" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.3.32 - - [27/Feb/2025:13:11:09 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.31+" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.3.32 - - [27/Feb/2025:13:11:14 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.31+" "-"
nginx-7c94c9bdcb-44hzp nginx 192.168.3.250 - - [27/Feb/2025:13:11:16 +0000] "GET / HTTP/1.1" 200 409 "-" "ELB-HealthChecker/2.0" "-"
...

 

nginx에 속한 모든 파드의 로그를 실시간으로 출력할 수 있다.

 

kubectl exec -it deploy/nginx -- ls -l /opt/bitnami/nginx/logs/
Defaulted container "nginx" out of: nginx, preserve-logs-symlinks (init)
total 0
lrwxrwxrwx. 1 1001 1001 11 Feb 27 13:10 access.log -> /dev/stdout
lrwxrwxrwx. 1 1001 1001 11 Feb 27 13:10 error.log -> /dev/stderr

 

액세스, 에러 로그가 저장되는 위치도 확인이 가능하다.

저장되는 위치가 접근 로그는 stdout, 에러 로그는 stderr로 전송 되는데 컨테이너 환경에서 배포되어 별도 파일로 로그를 떨구지 않기 때문이다. 따라서 표준 출력, 표준 에러를 활용하여 로그를 기록하는 것이다

 

---

Amazon CloudWatch & Fluent Bit

Fluent Bit은 로그 및 메트릭 데이터를 수집, 처리, 필터링 및 전송하는 경량 로그 프로세서로 쿠버네티스, Docker 등 여러 환경에서 로그 데이터를 효율적으로 수집하고 전송하는 역할을 수행한다.

저장된 로그는 aws 클라우드 워치를 통해 대시보드에서 확인이 가능하다.

 

export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)

aws eks create-addon --addon-name amazon-cloudwatch-observability --cluster-name myeks --service-account-role-arn arn:aws:iam::$ACCOUNT_ID:role/myeks-cloudwatch-agent-role

 

fluent bit 파드가 로그를 저장하는 방법

application-log.conf:
----
[INPUT]
  Name                tail
  Tag                 application.*
  Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  Path                /var/log/containers/*.log
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_container.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/fluent-bit*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_log.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/cloudwatch-agent*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_cwagent.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                aws
  Match               application.*
  az                  false
  ec2_instance_id     false
  Enable_Entity       true

[FILTER]
  Name                kubernetes
  Match               application.*
  Kube_URL            https://kubernetes.default.svc:443
  Kube_Tag_Prefix     application.var.log.containers.
  Merge_Log           On
  Merge_Log_Key       log_processed
  K8S-Logging.Parser  On
  K8S-Logging.Exclude Off
  Labels              Off
  Annotations         Off
  Use_Kubelet         On
  Kubelet_Port        10250
  Buffer_Size         0
  Use_Pod_Association On

[OUTPUT]
  Name                cloudwatch_logs
  Match               application.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  extra_user_agent    container-insights
  add_entity          true

 

fluent bit는 로그를 수집(INPUT)하고 필터링(FILTER)해서 aws로 전송(OUTPUT)하는 구조로 로그를 처리하는데,

CloudWatch 에이전트는 쿠버네티스 노드의 로그(/var/lib/docker, /run/containerd/containerd.sock)와 시스템 메트릭(/sys, /dev/disk/)을 수집하여 AWS CloudWatch로 전송하는 것을 알 수 있다.

 

kubectl describe -n amazon-cloudwatch ds cloudwatch-agent
Volumes:
...
   rootfs:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:

 

 

HostPath가 루트 디렉터리 전체를 마운트한 것을 확인할 수 있는데, 이는 파드가 노드의 모든 파일을 읽거나 변경할 수 있는 권한을 가질 수 있게 되어 위험한 설정이라는 것을 확인할 수 있다.

 

 

containerinsights 경로 밑으로 fluent bit에서 보낸 로그들이 생긴것을 확인할 수 있다. 

• application : 컨테이너와 파드 로그
• dataplane : 쿠버네티스 데이터플레인 로그
• performance : 성능 로그 이벤트

 

부하 테스트

ab -c 500 -n 30000 https://nginx.$MyDomain/

 

 

nginx 컨테이너에서 부하를 발생하면 애플리케이션 단 로그가 200으로 응답값이 제대로 발생하는 것을 확인할 수 있다.

 

 

로그 인사이트에서 애플리케이션 하위에 컨테이너에서 발생한 에러 로그를 확인할 수 있다.

 

대시보드

 

실습 완료 후 삭제

aws eks delete-addon --cluster-name $CLUSTER_NAME --addon-name amazon-cloudwatch-observability

 

실습 완료 후 애드온과 로그 그룹을 삭제!