카프카 클러스터 구성

목표

1. 도커 컴포즈 기반의 카프카 클러스터를 구성한다.
2. 3개의 브로커를 설정한다.
3. kafka-ui 를 사용하여 카프카 클러스터를 관리한다.
4. 프로듀서 테스트를 위한 도커 컨테이너를 추가로 설정한다.
   1. cmd
   2. java
   3. rust
5. 컨슈머 테스트를 위한 도커 컨테이너를 추가로 설정한다.
   1. cmd
   2. java
   3. rust
6. 최종 목표는 러스트를 사용하여 멀티쓰레드로 프로듀서와 컨슈머를 구현하는 것이다.

7. 참고 페이지

도커 컴포즈 설정

1. docker-compose.yml 파일을 생성한다.

2. 참고 사이트 설정을 기반으로 작성한다.

   docker-compose.yml

   networks:
     kafka_network:
   
   volumes:
     Kafka00:
       driver: local
     Kafka01:
       driver: local
     Kafka02:
       driver: local
   
   services:
     ##Kafka 00
     Kafka00Service:
       image: bitnami/kafka:3.5.1-debian-11-r72
       restart: unless-stopped
       container_name: Kafka00Container
       ports:
         - '10000:9094'
       environment:
         - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
         # KRaft settings
         - KAFKA_CFG_BROKER_ID=0
         - KAFKA_CFG_NODE_ID=0
         - KAFKA_KRAFT_CLUSTER_ID=HsDBs9l6UUmQq7Y5E6bNlw
         - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@Kafka00Service:9093,1@Kafka01Service:9093,2@Kafka02Service:9093
         - KAFKA_CFG_PROCESS_ROLES=controller,broker
         # Listeners
         - ALLOW_PLAINTEXT_LISTENER=yes
         - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
         - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://Kafka00Service:9092,EXTERNAL://127.0.0.1:10000
         - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
         - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
         - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
         # Clustering
         - KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=3
         - KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
         - KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=2
       networks:
         - kafka_network
       volumes:
         - 'Kafka00:/bitnami/kafka'
     ##Kafka 01
     Kafka01Service:
       image: bitnami/kafka:3.5.1-debian-11-r72
       restart: unless-stopped
       container_name: Kafka01Container
       ports:
         - '10001:9094'
       environment:
         - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
         # KRaft settings
         - KAFKA_CFG_BROKER_ID=1
         - KAFKA_CFG_NODE_ID=1
         - KAFKA_KRAFT_CLUSTER_ID=HsDBs9l6UUmQq7Y5E6bNlw
         - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@Kafka00Service:9093,1@Kafka01Service:9093,2@Kafka02Service:9093
         - KAFKA_CFG_PROCESS_ROLES=controller,broker
         # Listeners
         - ALLOW_PLAINTEXT_LISTENER=yes
         - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
         - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://Kafka01Service:9092,EXTERNAL://127.0.0.1:10001
         - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
         - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
         - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
         # Clustering
         - KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=3
         - KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
         - KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=2
       networks:
         - kafka_network
       volumes:
         - 'Kafka01:/bitnami/kafka'
     ##Kafka 02
     Kafka02Service:
       image: bitnami/kafka:3.5.1-debian-11-r72
       restart: unless-stopped
       container_name: Kafka02Container
       ports:
         - '10002:9094'
       environment:
         - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
         # KRaft settings
         - KAFKA_CFG_BROKER_ID=2
         - KAFKA_CFG_NODE_ID=2
         - KAFKA_KRAFT_CLUSTER_ID=HsDBs9l6UUmQq7Y5E6bNlw
         - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@Kafka00Service:9093,1@Kafka01Service:9093,2@Kafka02Service:9093
         - KAFKA_CFG_PROCESS_ROLES=controller,broker
         # Listeners
         - ALLOW_PLAINTEXT_LISTENER=yes
         - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
         - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://Kafka02Service:9092,EXTERNAL://127.0.0.1:10002
         - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
         - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
         - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
         # Clustering
         - KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=3
         - KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3
         - KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=2
       networks:
         - kafka_network
       volumes:
         - 'Kafka02:/bitnami/kafka'
   
     KafkaWebUiService:
       image: provectuslabs/kafka-ui:latest
       restart: unless-stopped
       container_name: KafkaWebUiContainer
       ports:
         - '8080:8080'
       environment:
         - KAFKA_CLUSTERS_0_NAME=Local-Kraft-Cluster
         - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=Kafka00Service:9092,Kafka01Service:9092,Kafka02Service:9092
         - DYNAMIC_CONFIG_ENABLED=true
         - KAFKA_CLUSTERS_0_AUDIT_TOPICAUDITENABLED=true
         - KAFKA_CLUSTERS_0_AUDIT_CONSOLEAUDITENABLED=true
         #- KAFKA_CLUSTERS_0_METRICS_PORT=9999
       networks:
         - kafka_network

3. 서비스를 실행한다.

   docker compose up -d

4. 브라우저에서 http://localhost:8080으로 접속하여 kafka-ui를 확인한다.