GitLab: Components, Architecture, Infrastructure, and Launching from the Helm Chart in Minikube

By | 02/02/2023
 

As GitLab recently changed the policy of providing free access, and now only 5 users will be available for the Free subscription, we decided to move to the self-hosted version.

In general, they have interesting terms with the license: the price depends on the number of users, you can buy it for at least one year, and after the purchase, you cannot reduce the number of users in the license (but you can increase it).

Our GitLab will live in Kubernetes, and there are many questions before launch, especially since I personally did not use GitLab very much before.

We will deploy GitLab through ArgoCD, will run it in AWS Elastic Kubernetes Service, and will use AWS S3 for the object store. But more on that later, first let’s see what GitLab is “from the inside” and how to deploy it in general.

GitLab Operator vs GitLab Helm chart

First – GitLab Operator or GitLab Helm chart?

I looked at the possibilities and didn’t see much difference between the Operator and the chart, moreover, the Operator uses the same Helm chart under the hood.

For backup and recovery capabilities, similar utilities are used and the process is the same, see General backup and restore guidance.

But with the version update process of the GitLab instance, the documentation of the Operator looks simpler. See Upgrade the GitLab chart vs. How the Operator handles GitLab upgrades.

And even Gitlab.com itself is deployed with their chart :

The largest known GitLab instance is on GitLab.com, which is deployed using our official GitLab Helm chart

But in the documentation for the Operator, the “Experimental” and “Beta” words are met too often, so for now probably let’s run without it.

In any case, everything is tied to the chart, so we will get acquainted mainly with it and its parameters.

GitLab architecture and components

For reference:

A simplified scheme from the documentation:

Here:

  • NGINX: accepting incoming connections
  • GitLab Workhorse: a reverse proxy for file uploads and downloads, Git push/pull, etc.
  • Puma: A Ruby web server used by GitLab for its API and web pages
  • Sidekiq: for creating and managing job queues
    • uses Redis to store job information
  • PostgreSQL: storage of information about users, access rights, metadata, etc.
  • GitLab Shell: works with repositories over SSH and manages access keys
    • calls Gitaly to process Git objects
    • sends information to Redis to create jobs in Sidekiq
  • Gitaly: Git RPC server, handles tasks in repositories received from GitLab Shell and GitLab web applications

Infrastructure

GitLab requires PostgreSQL, Redis, AWS S3 buckets, and mail service to receive and send emails.

Helm chart installs PostgreSQL and Redis by default, but for production, PostgreSQL is recommended to be installed separately, see Configure the GitLab chart with an external database and Configure PostgreSQL, although we will use the PostgreSQL Operator and run a PostgreSQL cluster in Kubernetes.

Similar requirements for Redis and Gitaly – it is also desirable to run them off the chart and not in a Kubernetes cluster. See Installing GitLab by using Helm. Instead of Redis, we will most likely use KeyDB, also through the Operator, also in Kubernetes.

The Gitaly deployment documentation says that for up to 500 users, a single virtual machine with Gitaly itself is sufficient. If 1000 or more users are planned, it is recommended to run Gitaly Cluster with Praefect. See Gitaly > High Availability. Considering the number of users we have, I don’t see the point of moving it to a separate EC2, so we will deploy it together with the chart, then we will look at its work and resources, and maybe will move it to a dedicated Kubernetes node.

Minio is installed in the chart to work with the object store, but again an external service such as AWS S3 is recommended for production. See Configure the GitLab chart with an external object storage.

Since we are on AWS and using AWS ALB Controller, it makes sense to disable NGINX Controller, see Customize the GitLab Ingress options.

There is a good example of infrastructure in the GitLab on AWS Partner Solution Deployment Guide, but this is for really big projects, and we will do it more simply. However, the scheme itself is interesting and useful for understanding the general concept of infrastructure planning:

In addition, there are Reference architectures, which describe options for running GitLab under different workloads. From them, we may be particularly interested in the Cloud native hybrid, which describes running in Kubernetes (hybrid – because it is recommended to run some services out of a Kubernetes cluster):

For LoadBalancer, the least outstanding requests are recommended instead of the standard round-robin – you must not forget.

It is useful to go through all the available options, see what and how you can configure, see GitLab Helm chart deployment options, we will deal with this in the next post.

GitLab monitoring is a separate topic, we will come to it later, for now, see Monitoring GitLab .

Running GitLab in Minikube

Generally used by GitLab’s developers who are working on features for Kubernetes, but we use it to familiarize ourselves with the GitLab chart. See Developing for Kubernetes with minikube.

Let’s start Minikube:

[simterm]

$ minikube start --cpus 4 --memory 10240

[/simterm]

Enable the Ingress plugin:

[simterm]

$ minikube addons enable ingress

[/simterm]

Clone the repository and install the dependencies – it is useful to look at them, although they are described in the documentation of the chart:

[simterm]

$ git clone https://gitlab.com/gitlab-org/charts/gitlab.git
$ cd gitlab
$ helm dependency update
...
Dependency gitlab did not declare a repository. Assuming it exists in the charts directory
Dependency certmanager-issuer did not declare a repository. Assuming it exists in the charts directory
Dependency minio did not declare a repository. Assuming it exists in the charts directory
Dependency registry did not declare a repository. Assuming it exists in the charts directory
Downloading cert-manager from repo https://charts.jetstack.io/
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading postgresql from repo https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
Downloading gitlab-runner from repo https://charts.gitlab.io/
Downloading grafana from repo https://grafana.github.io/helm-charts
Downloading redis from repo https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
Dependency nginx-ingress did not declare a repository. Assuming it exists in the charts directory
...

[/simterm]

Check the Minicube IP:

[simterm]

$ minikube ip
192.168.49.2

[/simterm]

And install with the values-minikube.yaml:

[simterm]

$ helm upgrade --install gitlab . --timeout 600s -f https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube.yaml --set global.hosts.domain=$(minikube ip).nip.io --set global.hosts.externalIP=$(minikube ip)

[/simterm]

Let’s check the pods – there are a lot of things here:

[simterm]

$ kubectl get pod
NAME                                          READY   STATUS      RESTARTS      AGE
gitlab-gitaly-0                               1/1     Running     0             11m
gitlab-gitlab-exporter-5c8dbdc954-hr7jj       1/1     Running     0             11m
gitlab-gitlab-runner-7c4488ff58-bg8f5         0/1     Running     3 (39s ago)   8m30s
gitlab-gitlab-shell-7f9f5bb9ff-qlpxr          1/1     Running     0             11m
gitlab-gitlab-shell-7f9f5bb9ff-wc9rx          1/1     Running     0             11m
gitlab-kas-84cb5c548b-jbp69                   1/1     Running     0             11m
gitlab-kas-84cb5c548b-jxqqr                   1/1     Running     0             11m
gitlab-migrations-2-vw5jd                     0/1     Completed   0             8m30s
gitlab-minio-74467697bb-z8nms                 1/1     Running     0             11m
gitlab-minio-create-buckets-2-b6zl6           0/1     Completed   0             8m30s
gitlab-postgresql-0                           2/2     Running     0             11m
gitlab-prometheus-server-6bf4fffc55-6xpfm     2/2     Running     0             11m
gitlab-redis-master-0                         2/2     Running     0             11m
gitlab-registry-cd64f65dc-frsmt               1/1     Running     0             8m30s
gitlab-registry-cd64f65dc-nvfv2               1/1     Running     0             8m10s
gitlab-sidekiq-all-in-1-v2-59ccd7d6b9-9mpmz   1/1     Running     0             8m30s
gitlab-toolbox-6586c478f5-ktj5x               1/1     Running     0             7m58s
gitlab-webservice-default-6787f4b5db-kfp9h    2/2     Running     0             6m55s
gitlab-webservice-default-6787f4b5db-q62f9    2/2     Running     0             8m30s

[/simterm]

Services:

[simterm]

$ kubectl get svc
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE
gitlab-gitaly                ClusterIP   None             <none>        8075/TCP,9236/TCP                     11m
gitlab-gitlab-exporter       ClusterIP   10.110.108.219   <none>        9168/TCP                              11m
gitlab-gitlab-shell          NodePort    10.110.231.31    <none>        32022:32022/TCP                       11m
gitlab-kas                   ClusterIP   10.102.124.129   <none>        8150/TCP,8153/TCP,8154/TCP,8151/TCP   11m
gitlab-minio-svc             ClusterIP   10.99.213.94     <none>        9000/TCP                              11m
gitlab-postgresql            ClusterIP   10.102.38.18     <none>        5432/TCP                              11m
gitlab-postgresql-headless   ClusterIP   None             <none>        5432/TCP                              11m
gitlab-postgresql-metrics    ClusterIP   10.107.123.35    <none>        9187/TCP                              11m
gitlab-prometheus-server     ClusterIP   10.104.133.44    <none>        80/TCP                                11m
gitlab-redis-headless        ClusterIP   None             <none>        6379/TCP                              11m
gitlab-redis-master          ClusterIP   10.96.133.252    <none>        6379/TCP                              11m
gitlab-redis-metrics         ClusterIP   10.105.193.58    <none>        9121/TCP                              11m
gitlab-registry              ClusterIP   10.109.96.193    <none>        5000/TCP                              11m
gitlab-webservice-default    ClusterIP   10.108.231.229   <none>        8080/TCP,8181/TCP,8083/TCP            11m
kubernetes                   ClusterIP   10.96.0.1        <none>        443/TCP                               23m

[/simterm]

Go to the https://gitlab.192.168.49.2.nip.io URL:

Find the password:

[simterm]

$ kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo
yNU***njU

[/simterm]

And log in as the root user :

Let’s see what we have deployed here:

  • gitaly – already known
  • gitlab-exporter – GitLab metrics collection for its Prometheus
  • gitlab-runner – workers for CI/CD
  • gitlab-shell – already known
  • kas – Kubernetes agent server для Gitlab Agent, “GitLab Agent for Kubernetes is an active in-cluster component for solving any GitLab<->Kubernetes integration tasks
  • migrations –  jobs for working with database migrations
  • minioHigh Performance Object Storage, API compatible with Amazon S3 – used when there is no S3
  • postgresql – already known
  • prometheus-server – GitLab monitoring
  • redis-master – Redis 🙂
  • registry – Container Registry for storing images
  • sidekiq – already known
  • toolbox – backups and other utilities
  • webservice-defaultGitLab Rails webserver

Let’s see what’s inside – go to the Admin:

Mmm… Delicious) And a lot. The administration of GitLab will have to be dealt with separately.

In the meantime, we can start investigating the Helm chart and try to deploy it in Kubernetes – see GitLab: Helm chart of values, dependencies, and deployment in Kubernetes with AWS S3.