사이드 프로젝트
디스크 이슈로 인한 서버 이전 회고
귀찮은 개발자
2024. 2. 11. 13:23
2년간 운영하던 서버의 디스크 160GB 까지 가득 차는 문제가 발생했다.
문제의 원인으로는 2가지가 있다.
1. Docker Registry 서버를 Private 으로 운영했기 때문에 Docker Image 가 너무 많았다.
2. SMTP 서버를 구축해 놓았기 때문에 Log 가 너무 많았다.
그간 디스크가 90프로 정도 찰 때마다 불필요한 이미지를 정리하고 logrotate 을 통해 로그를 관리하였고 마지막 영혼을 불태워 Docker Registry 서버를 가성비 좋은 S3 로 이전하려 했으나 디스크가 기다려주지 않아 서버를 확장하게 되었다.
아래는 마지막 순간까지 살아 있던 내 서비스들이다.
****@vps-*****:~$ ll
drwxrwxr-x 6 **** **** 4096 May 14 2023 capstone-2023-14/
-rw-rw-r-- 1 **** **** 7842 Aug 11 2022 ddagent-install.log
drwxrwxr-x 7 **** **** 4096 Jun 5 2022 docker-db/
drwxrwxr-x 3 **** **** 4096 Aug 7 2022 docker-jenkins/
drwxrwxr-x 6 **** **** 4096 Jun 5 2022 docker-registry-script/
drwxrwxr-x 4 **** **** 4096 Nov 8 2022 geol/
drwxrwxr-x 3 **** **** 4096 Mar 6 2023 join-us-new/
drwxrwxr-x 2 **** **** 4096 Aug 18 2022 monitoring/
drwxrwxr-x 3 **** **** 4096 Jun 25 2023 nginx-notion/
drwxrwxr-x 6 **** **** 4096 Oct 3 10:36 plan-b/
-rw-rw-r-- 1 **** **** 4056 Jun 20 2022 request-docs.php
drwxrwxr-x 7 **** **** 4096 Jun 14 2022 spring-boot-shopping-mall/
****@vps-*****:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f8df084cedbc plan-b_web "/docker-entrypoint.…" 4 months ago Up 3 months 80/tcp plan-b
95630a5ef61b docker.********.com/*****/main-service:latest "java -jar ssp.jar" 6 months ago Up 3 months main-service
5ba380236b86 docker.********.com/*****/chat-service:latest "java -jar chat.jar" 6 months ago Up 3 months chat-service
9bf6e90e1328 docker.********.com/*****/zookeeper:latest "/bin/sh -c '/usr/sb…" 6 months ago Up 3 months 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
cb9d33fdc72f docker.********.com/*****/rabbitmq:latest "docker-entrypoint.s…" 6 months ago Up 3 months 0.0.0.0:4369->4369/tcp, :::4369->4369/tcp, 0.0.0.0:5671-5672->5671-5672/tcp, :::5671-5672->5671-5672/tcp, 15691-15692/tcp, 0.0.0.0:15671-15672->15671-15672/tcp, :::15671-15672->15671-15672/tcp, 25672/tcp rabbitmq
dd6d665c98b1 docker.********.com/*****/apigate-service:latest "java -jar apigate.j…" 6 months ago Up 3 months 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp apigate-service
7ed3a7b65a89 docker.********.com/*****/discovery-service:latest "java -jar discovery…" 6 months ago Up 3 months 0.0.0.0:8761->8761/tcp, :::8761->8761/tcp discovery-service
2364e4ffc536 docker.********.com/*****/grafana:latest "/run.sh" 6 months ago Up 3 months 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafana
ea32dc68c383 docker.********.com/*****/prometheus:latest "/bin/prometheus --c…" 6 months ago Up 3 months 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
4607301c8e52 docker.********.com/*****/cp-server-connect-datagen:latest "/etc/confluent/dock…" 6 months ago Up 3 months 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 9092/tcp kafka-connect
7bbe2c6394c2 docker.********.com/*****/zipkin:latest "start-zipkin" 6 months ago Up 3 months (healthy) 9410/tcp, 0.0.0.0:9411->9411/tcp, :::9411->9411/tcp zipkin
0892d49d11fe docker.********.com/*****/config-service:latest "java -jar config.jar" 6 months ago Up 3 months 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp config-service
0c94222b4c45 frontend_capstone-frontend "/docker-entrypoint.…" 9 months ago Up 3 months 80/tcp capstone-frontend
35d966597a63 phpmyadmin/phpmyadmin "/docker-entrypoint.…" 9 months ago Up 3 months 80/tcp capstone-phpmyadmin
d3de080a99a1 quay.io/prometheus/node-exporter:latest "/bin/node_exporter …" 9 months ago Up 3 months 9100/tcp node-exporter
e50e871d601e redis "docker-entrypoint.s…" 10 months ago Up 3 months 0.0.0.0:4004->6379/tcp, :::4004->6379/tcp redis-backend
d6f5d101fa79 docker.********.com/***news:latest "docker-entrypoint.s…" 15 months ago Up 3 months 3000/tcp ***news
baaad40e0efd docker.********.com/******/***news-api:latest "apache2-foreground" 15 months ago Up 3 months 80/tcp ***news-api
7c3e96204c63 docker.********.com/throwback-thursday:latest "/docker-entrypoint.…" 15 months ago Up 3 months 80/tcp throwback-thursday
de35e51209a6 docker.********.com/******/***news:develop "docker-entrypoint.s…" 16 months ago Up 3 months ***news-develop
62176653b410 docker.********.com/northfacegawd/coffee-order:latest "docker-entrypoint.s…" 18 months ago Up 3 months coffee-order-web
aa44ca36563c docker.********.com/coffee-order/api:latest "apache2-foreground" 19 months ago Up 3 months 80/tcp coffee-order-api
e4cf8f49be0b php:7.1-apache "docker-***entrypoi…" 20 months ago Up 3 months 80/tcp geol
cb08d1d5fab2 phpmyadmin/phpmyadmin "/docker-entrypoint.…" 20 months ago Up 3 months 80/tcp spring-shopping-pwd
0a4fa77fec19 spring-boot-shopping-mall_spring-shopping-app "/usr/local/bin/mvn-…" 20 months ago Up 3 months spring-shopping-app
3d41eadc3568 mysql:5.7 "docker-entrypoint.s…" 20 months ago Up 3 months 3306/tcp, 33060/tcp spring-shopping-db
7360c90ad77d phpmyadmin/phpmyadmin:5.0.4 "/docker-entrypoint.…" 20 months ago Up 3 months 80/tcp phpmyadmin
328fbba91dd4 mysql:8.0.29 "docker-entrypoint.s…" 20 months ago Up 3 months 3306/tcp, 33060/tcp mysql
00828657b41d 53aa81e8adfa "docker-entrypoint.s…" 20 months ago Up 3 months 6379/tcp redis
ec502e17d68a postgres:13.3 "docker-entrypoint.s…" 20 months ago Up 3 months 5432/tcp postgres
42547a9ed59c jc21/nginx-proxy-manager:2.9.18 "/init" 20 months ago Up 3 months 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 81/tcp nginx-app
18a8f2f0b839 jc21/mariadb-aria:10.4.15 "/scripts/run.sh" 20 months ago Up 3 months 3306/tcp nginx-db
3d22db02d9de hyper/docker-registry-web:latest "start.sh" 20 months ago Up 3 months 8080/tcp registry-web
268e7b9a4df8 portainer/portainer-ce:2.13.1 "/portainer" 20 months ago Up 3 months 8000/tcp, 9000/tcp, 9443/tcp portainer
7bf1acf3f6fc registry:2.8.1 "/entrypoint.sh /etc…" 20 months ago Up 3 months 5000/tcp registry
계획
- 서버는 총 2대로 확장할 생각이며
- Docker Swrm 을 추가할 계획이다.
- Docker Registry 는 당연히 Harbor + S3 을 사용해 Private 하게 갈 생각이다.
- Database 는 Read-write, Read-only 를 지원할 것이며 중복되는 Database 는 실행하지 않 것이다.
- Docker Image 을 다시 빌드하고
- 대부분 프로젝트는 단일 컨테이너 이기 때문에 Image 에 따라 1개씩 디폴로이 할 수 있도록 Script 을 추가할 것이고 MSA 프로젝트의 경우 docker compose 으로 일부 작성 되어 있기 때문에 docker stack 으로 관리할 것이다.
서버를 한대씩 추가할 때 마다 아래 스크립트르 통해 기본적으로 필요한 것들을 추가해 나갈 생각이지만
앞으로 2년간은 2대면 충분할 것 같다.
# Init SSH
$ echo ssh-rsa ~~~ >> ~/.ssh/authorized_keys
# Update OS
$ do-release-upgrade
# Install Docker
$ sudo apt-get -y update
$ sudo apt -y update
$ sudo apt install -y docker.io docker-compose
$ sudo usermod -aG docker ${USER}
$ sudo service docker start
# Init ZSH
$ sudo apt install -y zsh
# which zsh 의 경로
$ sudo chsh -s /usr/bin/zsh
$ sudo reboot
$ login
This is the Z Shell configuration function for new users,
zsh-newuser-install.
You are seeing this message because you have no zsh startup files
(the files .zshenv, .zprofile, .zshrc, .zlogin in the directory
~). This function can help you with a few settings that should
make your use of the shell easier.
You can:
(q) Quit and do nothing. The function will be run again next time.
(0) Exit, creating the file ~/.zshrc containing just a comment.
That will prevent this function being run again.
(1) Continue to the main menu.
(2) Populate your ~/.zshrc with the configuration recommended
by the system administrator and exit (you will need to edit
the file by hand, if so desired).
--- Type one of the keys in parentheses ---
$ 2
$ echo $SHELL
```
# Setup .zshrc
$ sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
$ git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting
$ git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions
$ sudo apt install -y fonts-powerline
$ vim .zshrc
## 플러그인 추가 :zsh-syntax-highlighting, zsh-autosuggestions
## 테마 수정
$ source .zshrc
# Install Vim
$ sudo apt install -y vim
$ git clone --depth=1 https://github.com/amix/vimrc.git ~/.vim_runtime
$ sh ~/.vim_runtime/install_awesome_vimrc.sh
# Docker Swarm Init & Join
$ docker swarm init --advertise-addr 서버IP
$ sudo ufw allow 2377
$ sudo ufw enable
$ docker swarm join-token -q worker
# Docker Swarm Test
$ docker stack deploy -c docker-compose.yml init-node
$ docker stack deploy -c docker-compose.yml init-db