Gitlab CI/CD 실전예제

저번에 포스팅한 Gitlab runner에 이어서 .gitlab-ci.yml 파일 작성하는 법에 대해 알아보겠다.

 

이전에 작성한 Circle CI에서는 nginx를 docker를 통해서 띄웠다.

이 방식은 불필요하게 프로젝트마다 nginx가 설치되고 nginx 이미지가 동일하기때문에 자칫 잘못하면 nginx가 한번에 다운될 수 도 있다.

이번에는 nginx를 컨테이너 외부에 설치했다.

 

이전 포스팅에서는 gitlab-runner를 java 버전별로 설치했는데 서버 성능 이슈때문에 병렬로 pipeline을 실행하지 못하게 하나의 runner만 등록하게 변경했다.

현재 runner에는 모든 java버전이 설치되어 있다.

 

 

.gitlab-ci.yml

docker login시 ~/.docker/config.json 에 로그인 정보가 남으므로 서버에서 로그인을 했다면 before_script는 굳이 작성할 필요가 없다.

 

그 외 variables를 통해 프로젝트 맞춤 세팅을 할 수가 있다. (NGINX_PORT에 프로젝트 포트를 써줘야한다.)

 

runner에 모든 Java 버전이 설치되어 있으므로 JAVA_HOME 환경변수를 통해 Java 버전을 관리한다.

 

ssh서버 접속을 위해 미리 TARGET_SERVER의 ~/.ssh/authorized_keys에 public key를 등록해 두어야 한다.
StrictHostKeyChecking=no 옵션 대신 runner 컨테이너 내부의 ~/.ssh/known_hosts에 TARGET_SERVER를 등록하는 것이 정석이지만 docker 특성상 재부팅이 정보가 사라지기 때문에 부득이 하게 해당 옵션을 사용했다.

workflow:
  # master 브랜치나 main 브랜치에 pull request를 merge 했을 때만 CI/CD 동작
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push"'
#    - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master" || $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'

default:
  # 실행할 runner 선택
  tags:
    - java

# CI/CD 가 동작 하는 단계
stages:
  - build & push
  - deploy

  # 모든 스크립트가 실행 되기 전에 동작
before_script:
  - echo "$DOCKER_HUB_PASSWORD" | docker login -u "$DOCKER_HUB_USERNAME" --password-stdin && echo "Docker Login Successful" || echo "Docker Login Failed"

# 변수 세팅
variables:
  TARGET_HOST: $SERVER223_HOST
  TARGET_SERVER: $SERVER223_IP
  BUILD_NUMBER: $CI_PIPELINE_ID
  REPO_NAME: $CI_PROJECT_NAME
  PROJECT_DIR: "/home/docker-app/$REPO_NAME"
  BLUE_PORT: $BLUE_PORT
  GREEN_PORT: $GREEN_PORT
  NGINX_PORT: $NGINX_PORT
  DOWN_DELTA_TIME: "300"
  HEALTH_CHECK_PATH: ""


  # java project build
build & push:
  stage: build & push
  script:
    - export JAVA_HOME=$PATH_JAVA17
    
    - ./gradlew build --exclude-task test --no-daemon
    - docker build -t rm/$REPO_NAME/$REPO_NAME:local .

    # docker hubub에 올릴 이미지 태그 지정 및 푸시
    - docker tag rm/$REPO_NAME/$REPO_NAME:local rm/$REPO_NAME/$REPO_NAME:$BUILD_NUMBER
    - docker tag rm/$REPO_NAME/$REPO_NAME:local rm/$REPO_NAME/$REPO_NAME:latest
    - docker push rm/$REPO_NAME/$REPO_NAME:$BUILD_NUMBER
    - docker push rm/$REPO_NAME/$REPO_NAME:latest

# 배포 진행
deploy:
  stage: deploy
  script:
    - set -e # 첫 번째 에러에서 스크립트 중지

    # key 등록
    - |
      if [ -f ~/.ssh/id_rsa ]; then
      echo "id_rsa exists."
      else
      mkdir -p ~/.ssh
      
      echo "${SERVER_PRIVATE_KEY}" > ~/.ssh/id_rsa
      echo "${SERVER_PUBLIC_KEY}" > ~/.ssh/id_rsa.pub
      chmod 600 ~/.ssh/id_rsa
      chmod 644 ~/.ssh/id_rsa.pub
      fi

    - ssh -o StrictHostKeyChecking=no $TARGET_HOST@$TARGET_SERVER "sudo mkdir -p $PROJECT_DIR/sh && sudo chmod 777 $PROJECT_DIR/sh"
    - scp -o StrictHostKeyChecking=no "$CI_PROJECT_DIR/deploy.sh" $TARGET_HOST@$TARGET_SERVER:$PROJECT_DIR/sh/deploy.sh
    - ssh -o StrictHostKeyChecking=no -T $TARGET_HOST@$TARGET_SERVER "set -e;sudo bash $PROJECT_DIR/sh/deploy.sh $REPO_NAME $PROJECT_DIR $TARGET_SERVER $BUILD_NUMBER $BLUE_PORT $GREEN_PORT $NGINX_PORT $DOWN_DELTA_TIME $HEALTH_CHECK_PATH"

 

deploy.sh

#!/bin/bash

REPO_NAME=$1
DIR_PROJECT=$2
TARGET_SERVER=$3
BUILD_NUMBER=$4
BLUE_PORT=$5
GREEN_PORT=$6
NGINX_PORT=$7
DOWN_DELTA_TIME=$8
HEALTH_CHECK_PATH=$9


echo "===================="
# 실행중인 백그라운드 종료
if [ -f "$DIR_PROJECT/sh/down_live_after_delay.pid" ]; then
    OLD_PID=$(cat "$DIR_PROJECT/sh/down_live_after_delay.pid")
    if ps -p "$OLD_PID" > /dev/null; then
        echo "Terminating existing background process..."
        kill -9 "$OLD_PID"
    else
        echo "No existing background process found with PID $OLD_PID"
    fi
else
    echo "No existing background process found"
fi

echo "===================="
echo "REPO_NAME: $REPO_NAME"
echo "BUILD_NUMBER: $BUILD_NUMBER"
echo "TARGET_SERVER: $TARGET_SERVER"
echo "BLUE_PORT: $BLUE_PORT"
echo "GREEN_PORT: $GREEN_PORT"
echo "NGINX_PORT: $NGINX_PORT"
echo "DIR_PROJECT: $DIR_PROJECT"
echo "DOWN_DELTA_TIME: $DOWN_DELTA_TIME"
echo "HEALTH_CHECK_PATH: $HEALTH_CHECK_PATH"


if [ -z "$REPO_NAME" ] || [ -z "$BUILD_NUMBER" ] || [ -z "$BLUE_PORT" ] || [ -z "$GREEN_PORT" ] || [ -z "$NGINX_PORT" ] || [ -z "$DIR_PROJECT" ] || [ -z "$DOWN_DELTA_TIME" ]; then
    echo "One or more variables are empty."
    exit 1
fi


echo "===================="
if [ -f "/etc/nginx/conf.d/$REPO_NAME.conf" ]; then
NOW_PORT=$(grep 'proxy_pass' "/etc/nginx/conf.d/$REPO_NAME.conf" | awk -F: '{print $3}' | awk -F/ '{print $1}' | awk '{sub(/;$/, "", $0); print $0}')
else
cat << EOF > "/etc/nginx/conf.d/$REPO_NAME.conf"
server {
    listen ${NGINX_PORT};
    server_name 127.0.0.1 localhost ${TARGET_SERVER};

    location / {
        proxy_pass http://127.0.0.1:${BLUE_PORT};
    }

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
}
EOF
NOW_PORT=$GREEN_PORT
fi

echo "NOW_PORT: $NOW_PORT"

echo "===================="
if [ "$NOW_PORT" == "$GREEN_PORT" ]
then
  LIVE_PORT=$GREEN_PORT
  LIVE_COLOR="green"
  TARGET_PORT=$BLUE_PORT
  TARGET_COLOR="blue"
else
  LIVE_PORT=$BLUE_PORT
  LIVE_COLOR="blue"
  TARGET_PORT=$GREEN_PORT
  TARGET_COLOR="green"
fi

cd "$DIR_PROJECT" || exit 1

echo "===================="
echo "create-docker-compose.sh"
OLD_BUILD_NUMBER=$((BUILD_NUMBER - 1))
cat << EOF > "$DIR_PROJECT/docker-compose.yml"
version: '3.3'

services:
  ${TARGET_COLOR}:
    image: rm/${REPO_NAME}/${REPO_NAME}:latest
    container_name: ${REPO_NAME}-${TARGET_COLOR}
    ports:
      - "${TARGET_PORT}:${NGINX_PORT}"
    volumes:
      - ${DIR_PROJECT}/logs:/logs
    environment:
      - TZ=Asia/Seoul
  ${LIVE_COLOR}:
    image: rm/${REPO_NAME}/${REPO_NAME}:${OLD_BUILD_NUMBER}
    container_name: ${REPO_NAME}-${LIVE_COLOR}
    ports:
     - "${LIVE_PORT}:${NGINX_PORT}"
    volumes:
     - ${DIR_PROJECT}/logs:/logs
    environment:
     - TZ=Asia/Seoul
EOF

echo "===================="

echo "now state"
Target=$(curl --write-out '%{http_code}' --silent --output /dev/null "127.0.0.1:$TARGET_PORT/$HEALTH_CHECK_PATH")
Live=$(curl --write-out '%{http_code}' --silent --output /dev/null "127.0.0.1:$LIVE_PORT/$HEALTH_CHECK_PATH")

echo "API Target: $Target"
echo "API Live: $Live"


# Deploy the new version to the new environment
echo "===================="
echo "docker compose up"
docker-compose -f "$DIR_PROJECT/docker-compose.yml" pull ${TARGET_COLOR} || exit 1
docker-compose -f "$DIR_PROJECT/docker-compose.yml" up -d ${TARGET_COLOR} || exit 1
Target=$(curl --write-out '%{http_code}' --silent --output /dev/null "127.0.0.1:$TARGET_PORT"/$HEALTH_CHECK_PATH)
Live=$(curl --write-out '%{http_code}' --silent --output /dev/null "127.0.0.1:$LIVE_PORT/$HEALTH_CHECK_PATH")

echo "API Target: $Target"
echo "API Live: $Live"


echo "===================="
run_tests() {
  # shellcheck disable=SC2034
  for i in {1..10}; do
    Target=$(curl --write-out '%{http_code}' --silent --output /dev/null "127.0.0.1:$TARGET_PORT/$HEALTH_CHECK_PATH")
    Live=$(curl --write-out '%{http_code}' --silent --output /dev/null "127.0.0.1:$LIVE_PORT/$HEALTH_CHECK_PATH")

    echo "API Target: $Target"
    echo "API Live: $Live"

    if [ "$Target" -eq 200 ]; then
      return 0
    fi
    echo "API is not ready yet cnt: [$i/10]]"
    # Wait for 6 seconds before the next try
    sleep 6
  done

  echo "API has not become ready within 60 seconds"
  # If the API did not respond with 200 OK within 60 seconds, return a failure code
  return 1
}

if run_tests; then
  # If Lives passed, switch the traffic to the new environment
  echo "Success!! Change Target: $TARGET_COLOR"
  NGINX_TARGET_PORT=$TARGET_PORT
  IS_SUCCESS=0

else
  # If Lives failed, stop the new environment
  echo "Fail.. Change LIVE: $LIVE_COLOR"
  NGINX_TARGET_PORT=$LIVE_PORT
  IS_SUCCESS=1
fi

echo "===================="
current_port=$(grep 'proxy_pass' "/etc/nginx/conf.d/$REPO_NAME.conf" | awk -F: '{print $3}' | awk -F/ '{print $1}' | awk -F\; '{print $1}')

if [ "$current_port" -eq "$NGINX_TARGET_PORT" ]; then
echo "same port"
exit 1
else
sed -i "s/proxy_pass http:\/\/127.0.0.1:$current_port;/proxy_pass http:\/\/127.0.0.1:$NGINX_TARGET_PORT;/" "/etc/nginx/conf.d/$REPO_NAME.conf"
# Nginx 설정 검사
output=$(sudo nginx -t 2>&1)

# "success" 문구가 있는지 확인
if [[ $output == *"test is successful"* ]]; then
  # 설정이 올바르면 Nginx를 리로드
  sudo nginx -s reload
else
  # 설정에 문제가 있으면 에러 메시지 출력
  echo "Nginx configuration test failed"
  echo "$output"
  exit 1
fi

fi

echo "===================="


if [ $IS_SUCCESS ]; then
  if docker ps --format '{{.Names}}' |  grep "$REPO_NAME-$LIVE_COLOR"; then

cat << EOF > "$DIR_PROJECT/sh/down_live_after_delay.sh"
#!/bin/bash

sleep $DOWN_DELTA_TIME
echo "Down Live after delay: $LIVE_COLOR"
docker stop $REPO_NAME-$LIVE_COLOR
EOF

    # 실행 권한 부여
    chmod +x "$DIR_PROJECT/sh/down_live_after_delay.sh"

    # nohup으로 실행
    nohup "$DIR_PROJECT/sh/down_live_after_delay.sh" > /dev/null 2>&1 &

    # 백그라운드 작업의 PID를 파일에 저장
    echo $! > "$DIR_PROJECT/sh/down_live_after_delay.pid"
    echo "Down Live after delay: $LIVE_COLOR, PID: $!, $DOWN_DELTA_TIME"
  else
    echo "Not Found Live: $LIVE_COLOR"
  fi
else
  if docker ps --format '{{.Names}}' |  grep "$REPO_NAME-$TARGET_COLOR"; then
    echo "Down Target: $TARGET_COLOR"
    docker stop "$REPO_NAME-$TARGET_COLOR"
  else
    echo "Not Found Target: $TARGET_COLOR"
  fi
fi
echo "===================="