Sentry: running self-hosted errors tracking system on an AWS EC2

By | 05/18/2019
0 Comment
 

Previously we used cloud-based Sentry version but then reached emails limit and our backend-team left without those notifications which are critical for their work.

A self-hosted version was planned a long time ago so now we have a chance to spin it up.

The post below describes how to start self-hosted Sentry on an AWS EC2 instance, configure email and test if from a Python.

Will use Sentry on-premise repository here.

Run AWS EC2, configure SSL from Let’s Encrypt, install NGINX, Docker and Docker Compose.

Those steps are described in details in the Bitwarden: an organization’s password manager self-hosted version installation on an AWS EC2 post, so I’ll skip them here.

Use EC2 type t3.medium as Sentry requires 3 GiB memory as minimal.

Running Sentry

Clone repository:

[simterm]

root@bttrm-sentry:/home/admin# mkdir /opt/sentry
root@bttrm-sentry:/home/admin# cd /opt/sentry/
root@bttrm-sentry:/opt/sentry# git clone https://github.com/getsentry/onpremise.git
root@bttrm-sentry:/opt/sentry# cd onpremise/

[/simterm]

Create Docker Volumes for data and PostgreSQL database:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# docker volume create --name=sentry-data && docker volume create --name=sentry-postgres
sentry-data
sentry-postgres

[/simterm]

Create config-file:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# cp -n .env.example .env

[/simterm]

Build images:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# docker-compose build
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
Building web
Step 1/1 : FROM sentry:9.1-onbuild
9.1-onbuild: Pulling from library/sentry
...
Successfully built 4840fec904c8
Successfully tagged onpremise_worker:latest

[/simterm]

Generate SECRET_KEY:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# docker-compose run --rm web config generate-secret-key
Creating network "onpremise_default" with the default driver
Pulling smtp (tianon/exim4:)...
latest: Pulling from tianon/exim4
...
Status: Downloaded newer image for tianon/exim4:latest
Pulling memcached (memcached:1.5-alpine)...
1.5-alpine: Pulling from library/memcached
...
Status: Downloaded newer image for memcached:1.5-alpine
Pulling redis (redis:3.2-alpine)...
3.2-alpine: Pulling from library/redis
...
Status: Downloaded newer image for redis:3.2-alpine
Pulling postgres (postgres:9.5)...
9.5: Pulling from library/postgres
...
Creating onpremise_smtp_1      ... done
Creating onpremise_postgres_1  ... done
Creating onpremise_memcached_1 ... done
Creating onpremise_redis_1     ... done
y0%***1yz

[/simterm]

The last string in the output – с y0%***1yz – is our key.

Edit the .env file and set this SECRET_KEY.

Populate database and create an admin user:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# docker-compose run --rm web upgrade
Starting onpremise_redis_1    ... done
Starting onpremise_postgres_1  ... done
Starting onpremise_memcached_1 ... done
Starting onpremise_smtp_1      ... done
12:06:24 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
12:06:27 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Creating tables ...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table south_migrationhistory
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
Migrating...
Running migrations for sentry:
 - Migrating forwards to 0472_auto__add_field_sentryapp_author.
...
Created internal Sentry project (slug=internal, id=1)

Would you like to create a user account now? [Y/n]: y
Email: [email protected]
Password: 
Repeat for confirmation: 
Should this user be a superuser? [y/N]: Y
User created: [email protected]
Added to organization: sentry
...

[/simterm]

Run the stack:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# docker-compose up 
onpremise_redis_1 is up-to-date
onpremise_memcached_1 is up-to-date
onpremise_postgres_1 is up-to-date
onpremise_smtp_1 is up-to-date
Creating onpremise_worker_1 ... done
Creating onpremise_web_1    ... done
Creating onpremise_cron_1   ... done
...

[/simterm]

Check containers:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# docker ps

[/simterm]

Open a browser and check if Sentry is working:

Log in, here you can configure email but I did it later using AWS SES:

Press Next, check it’s working:

Not sure what’s the worker message here, but it disappeared later.

Now, stop containers and create systemd unit-file – /etc/systemd/system/sentry.service:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# systemctl edit --force sentry

[/simterm]

Set here:

[Unit]
Description=Sentry service
Requires=docker.service
After=docker.service

[Service]
Restart=always
WorkingDirectory=/opt/sentry/onpremise

# Compose up
ExecStart=/usr/local/bin/docker-compose -f docker-compose.yml up

# Compose down, remove containers and volumes
ExecStop=/usr/local/bin/docker-compose -f docker-compose.yml down -v

[Install]
WantedBy=multi-user.target

In case of errors like:

May 18 12:38:33 bttrm-sentry systemd[1]: sentry.service: Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.

Add one more ExecStart, just empty:

[Unit]
Description=Sentry service
Requires=docker.service
After=docker.service

[Service]
Restart=always
WorkingDirectory=/opt/sentry/onpremise

# Compose up
ExecStart=
ExecStart=/usr/local/bin/docker-compose -f docker-compose.yml up

# Compose down, remove containers and volumes
ExecStop=/usr/local/bin/docker-compose -f docker-compose.yml down -v

[Install]
WantedBy=multi-user.target

Didn’t saw such the error earlier, just googled this solution here>>>.

Run the service:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# systemctl start sentry

[/simterm]

Check its status:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# systemctl status sentry
● sentry.service - Sentry service
   Loaded: loaded (/etc/systemd/system/sentry.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/sentry.service.d
           └─override.conf
   Active: active (running) since Sat 2019-05-18 12:39:21 UTC; 12min ago
 Main PID: 9209 (docker-compose)
      CPU: 643ms
   CGroup: /system.slice/sentry.service
           ├─9209 /usr/local/bin/docker-compose -f docker-compose.yml up
           └─9210 /usr/local/bin/docker-compose -f docker-compose.yml up

May 18 12:44:24 bttrm-sentry docker-compose[9209]: redis_1      | 1:M 18 May 12:44:24.080 * 100 changes in 300 seconds. Saving...
May 18 12:44:24 bttrm-sentry docker-compose[9209]: redis_1      | 1:M 18 May 12:44:24.081 * Background saving started by pid 11
May 18 12:44:24 bttrm-sentry docker-compose[9209]: redis_1      | 11:C 18 May 12:44:24.086 * DB saved on disk
May 18 12:44:24 bttrm-sentry docker-compose[9209]: redis_1      | 11:C 18 May 12:44:24.086 * RDB: 0 MB of memory used by copy-on-write
May 18 12:44:24 bttrm-sentry docker-compose[9209]: redis_1      | 1:M 18 May 12:44:24.181 * Background saving terminated with success
May 18 12:49:25 bttrm-sentry docker-compose[9209]: redis_1      | 1:M 18 May 12:49:25.066 * 100 changes in 300 seconds. Saving...
May 18 12:49:25 bttrm-sentry docker-compose[9209]: redis_1      | 1:M 18 May 12:49:25.066 * Background saving started by pid 12
May 18 12:49:25 bttrm-sentry docker-compose[9209]: redis_1      | 12:C 18 May 12:49:25.072 * DB saved on disk
May 18 12:49:25 bttrm-sentry docker-compose[9209]: redis_1      | 12:C 18 May 12:49:25.073 * RDB: 0 MB of memory used by copy-on-write
May 18 12:49:25 bttrm-sentry docker-compose[9209]: redis_1      | 1:M 18 May 12:49:25.167 * Background saving terminated with success

[/simterm]

Okay, all good.

Add to autostart:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# systemctl enable sentry

[/simterm]

Email

Email configuration documantation is available her>>>.

Edit docker-compose.yml, set variables, here is AWS SES example:

...
    SENTRY_EMAIL_HOST: email-smtp.us-east-1.amazonaws.com
    SENTRY_EMAIL_PORT: 587
    SENTRY_EMAIL_PASSWORD: BH3***gpM
    SENTRY_EMAIL_USER: AKI***OAQ
    SENTRY_EMAIL_USE_TLS: "true"
    SENTRY_SERVER_EMAIL: [email protected]
...

Pay attention to quotes around the “true”:

SENTRY_EMAIL_USE_TLS: “true”

Save and restart Sentry service:

[simterm]

root@bttrm-sentry:/opt/sentry/onpremise# systemctl restart sentry

[/simterm]

Python Sentry

Create a new project:

Get token:

Check from a workstation – install sentry-sdk:

[simterm]

[setevoy@setevoy-arch-work ~ ] $ sudo pip install sentry-sdk
[setevoy@setevoy-arch-work ~ ] $ python
...
>>> import sentry_sdk
>>> sentry_sdk.init("https://96e***[email protected]/2")
<sentry_sdk.hub._InitGuard object at 0x7efd4c487128>
>>> sentry_sdk.capture_message("Hello World")
'736347c534cb4e3eb27fb3e3c0641439'

[/simterm]

Check events in Sentry

And email:

Done.