Monit: мониторинг и перезапуск NGINX

Автор: | 11/08/2017

Домашняя страница проекта – тут>>>, документация – тут>>>.

Имеется сервис с двумя виртуальными машинами в Azure VMSS. Если для Dev ещё можно использовать решение типа C: “мониторинг” NGINX с помощью AF_INET, то для Production – используем Monit.

Пример тестовой установки в NGINX в Vagrant-боксе.

Подготовка

Запускаем:

[simterm]

$ cd /home/setevoy/VMs/Vagrant/nginx/
$ vagrant init obihann/nginx; vagrant up --provider virtualbox
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'obihann/nginx' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'obihann/nginx'
    default: URL: https://vagrantcloud.com/obihann/nginx
==> default: Adding box 'obihann/nginx' (v0.0.1) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/obihann/boxes/nginx/versions/0.0.1/providers/virtualbox.box
...

[/simterm]

Подключаемся:

[simterm]

$ vagrant ssh
...
vagrant@precise64:~$ sudo -s
root@precise64:~#
root@precise64:~# apt-get update && apt-get -y upgrade

[/simterm]

Установка Monit

Устанавливаем:

[simterm]

root@precise64:~# apt-get install monit vim curl
...
Setting up monit (1:5.3.2-1) ...
 * Starting daemon monitor monit

[/simterm]

Проверяем:

[simterm]

root@precise64:~# ps aux | grep monit
root      1243  0.0  0.3  30508  1184 ?        S    10:28   0:00 /usr/bin/monit -c /etc/monit/monitrc
root      1522  0.0  0.2  11692   956 pts/0    S+   10:29   0:00 grep --color=auto monit

[/simterm]

Либо:

[simterm]

root@precise64:~# service monit status
 * monit is running

[/simterm]

Статус:

[simterm]

root@precise64:~# monit status
monit: error connecting to the monit daemon

[/simterm]

Упс…

Файл настроек monit/etc/monit/monitrc, редактируем его, убираем комментарии в блоке:

...
## Monit has an embedded web server which can be used to view status of 
## services monitored and manage services from a web interface. See the
## Monit Wiki if you want to enable SSL for the web server. 
#
set httpd port 2812 and
   use address localhost  # only accept connection from localhost
   allow localhost        # allow localhost to connect to the server and
   allow admin:monit      # require user 'admin' with password 'monit'
   allow @monit           # allow users of group 'monit' to connect (rw)
   allow @users readonly  # allow users of group 'users' to connect readonly
...

Перезапускаем демон:

[simterm]

root@precise64:~# service monit restart
 * Stopping daemon monitor monit [ OK ]
 * Starting daemon monitor monit [ OK ]

[/simterm]

Проверяем:

[simterm]

root@precise64:~# netstat -lpn | grep 2812
tcp        0      0 127.0.0.1:2812          0.0.0.0:*               LISTEN      1922/monit

[/simterm]

Статус:

[simterm]

root@precise64:~# monit status
The Monit daemon 5.3.2 uptime: 0m 

System 'system_precise64'
  status                            Running
  monitoring status                 Monitored
  load average                      [0.03] [0.09] [0.06]
  cpu                               0.0%us 0.0%sy 0.0%wa
  memory usage                      68652 kB [18.3%]
  swap usage                        0 kB [0.0%]
  data collected                    Fri, 11 Aug 2017 10:36:35

[/simterm]

И статус мониторинга – пока тут пусто:

[simterm]

root@precise64:~# monit summary
The Monit daemon 5.3.2 uptime: 0m 

System 'system_precise64'           Running

[/simterm]

Monit NGINX

Проверяем pid-файл NGINX:

[simterm]

root@precise64:~# file /run/nginx.pid
/run/nginx.pid: ASCII text

[/simterm]

Проверяем сам NGINX:

[simterm]

root@precise64:~# curl -I localhost:80
HTTP/1.1 200 OK
...

[/simterm]

Для мониторинга NGINX – создаём файл настроек /etc/monit/conf.d/nginx:

check process nginx with pidfile /run/nginx.pid
    start program = "/usr/sbin/service nginx start" with timeout 60 seconds
    stop program  = "/usr/sbin/service nginx stop"
    if failed host 127.0.0.1 port 80 protocol http for 3 cycles then restart

Тут monit проверяет наличие pid-файла NGINX и если он не найден – то попробует запустить его, используя параметр start program (бокс – Ubuntu 14, с initd).

Затем – monit проверит доступность сервиса на 127.0.0.1:80, и в случае трёх неудачных попыток – попробует перезапустить его.

Если используется SSL – то меняем 80 на 443.

Проверяем синтаксис monit:

[simterm]

root@precise64:~# monit -t

[/simterm]

В случае ошибок – будет выдано сообщение вида:

[simterm]

root@precise64:~# monit -t
/etc/monit/monitrc:249: Error: syntax error ''

[/simterm]

Перегружаем monit:

[simterm]

root@precise64:~# service monit reload
 * Reloading daemon monitor configuration monit

[/simterm]

Проверяем сервисы:

[simterm]

root@precise64:~# monit summary
The Monit daemon 5.3.2 uptime: 0m 

Process 'nginx'                     Running
System 'system_precise64'           Running

[/simterm]

Для проверки – убиваем NGINX:

[simterm]

root@precise64:~# ps aux | grep "nginx: master" | grep -v grep
root      1352  0.0  0.3  62924  1332 ?        Ss   10:29   0:00 nginx: master process /usr/sbin/nginx
# kill 1352

[/simterm]

Сервисы monit:

[simterm]

root@precise64:~# monit summary
The Monit daemon 5.3.2 uptime: 6m 

Process 'nginx'                     Does not exist
System 'system_precise64'           Running

[/simterm]

Лог monit – и видим запущенный NGINX:

[simterm]

root@precise64:~# tail -f /var/log/monit.log 
[UTC Aug 11 10:43:17] error    : monit daemon died
[UTC Aug 11 10:43:48] error    : monit: Status not available -- the monit daemon is not running
[UTC Aug 11 10:43:52] info     : Starting monit daemon with http interface at [localhost:2812]
[UTC Aug 11 10:43:52] info     : Starting monit HTTP server at [localhost:2812]
[UTC Aug 11 10:43:52] info     : monit HTTP server started
[UTC Aug 11 10:43:52] info     : 'system_precise64' Monit started
[UTC Aug 11 10:49:52] error    : 'nginx' process is not running
[UTC Aug 11 10:49:52] info     : 'nginx' trying to restart
[UTC Aug 11 10:49:52] info     : 'nginx' start: /usr/sbin/service

[/simterm]

Проверяем:

[simterm]

root@precise64:~# ps aux | grep "nginx: master" | grep -v grep
root      2354  0.0  0.3  62924  1332 ?        Ss   10:49   0:00 nginx: master process /usr/sbin/nginx
root@precise64:~# curl -I localhost
HTTP/1.1 200 OK
Server: nginx/1.1.19

[/simterm]

Monit веб-интерфейс

Можно использовать GUI, доступный через встроенный в monit веб-сервер.

Т.к. машина запущена в Vagrant – редактируем Vagrantfile, меняем настройки сети:

...
config.vm.network "public_network"
...

Перезапускаем машину:

[simterm]

$ vagrant reload
...
==> default: Available bridged network interfaces:
1) enp0s25
2) br-9bcb68cdf3ad
3) docker0
==> default: When choosing an interface, it is usually the one that is
==> default: being used to connect to the internet.
    default: Which interface should the network bridge to? 1
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: bridged
...

[/simterm]

Получаем IP Vagrant-бокса:

[simterm]

$ vagrant ssh -c "hostname -I"
10.0.2.15 10.11.100.147 
Connection to 127.0.0.1 closed.

[/simterm]

10.11.100.147 – “внешний” IP.

Редактируем /etc/monit/monitrc и меняем адрес/порт/пользователя, например.

...
set httpd port 2812 and
   use address 10.11.100.147   # only accept connection from localhost
   allow localhost        # allow localhost to connect to the server and
   allow admin:monit      # require user 'admin' with password 'monit'
   allow @monit           # allow users of group 'monit' to connect (rw)
   allow @users readonly  # allow users of group 'users' to connect readonly
...

Перезапускаем, проверяем:

[simterm]

root@precise64:~# monit -t && service monit reload
 * Reloading daemon monitor configuration monit

[/simterm]

Пробуем открыть в браузере – и получаем ERR_SOCKET_NOT_CONNECTED.

Лог monit:

[UTC Aug 11 11:02:18] error    : monit: Denied connection from non-authorized client [10.11.100.119]

Находим свой IP:

[simterm]

$ ip  a s enp0s25 | grep inet
    inet 10.11.100.119/24 brd 10.11.100.255 scope global dynamic enp0s25

[/simterm]

Обновляем allow правила в /etc/monit/monitrc:

...
   allow localhost        # allow localhost to connect to the server and
   allow 10.11.100.119
...

Проверяем, перезапускаем, пробуем:

Mail alarm

Для отправки уведомлений при проблемах –  в файл /etc/monit/conf.d/nginx добавляем set mailserver и set alert:

check process nginx with pidfile /run/nginx.pid
    start program = "/usr/sbin/service nginx start" with timeout 60 seconds
    stop program  = "/usr/sbin/service nginx stop"
    if failed host 127.0.0.1 port 80 protocol http for 3 cycles then restart
    set mailserver 127.0.0.1
    set alert [email protected]

Подробнее – в документации>>>.

Ссылки по теме

Use Monit to Monitor + Restart nginx Service Automatically

How To Install and Configure Monit

Configure Monit Reverse Proxy nginx Linux

monit config file (nginx, mysql, redis, tomcat)