After running our self-hosted GitLab in production, we faced a bug: during git clone/pull/push operations, the request sometimes hung for 1-2 minutes. It looked like some kind of “floating” bug, that is, it could normally work 5 times, and then hangs once. The issues gitlab-shell timeouts For example, one time git clone works well: [simterm]… Read More »
So, let’s continue our journey with migrating GitLab to Kubernetes. See previous parts: GitLab: Components, Architecture, Infrastructure, and Launching from the Helm Chart in Minikube GitLab: Helm chart of values, dependencies, and deployment in Kubernetes with AWS S3 GitLab: міграція даних з GitLab cloud та процес backup-restore у self-hosted версії в Kubernetes In general, everything is working,… Read More »
Good – we learned how to launch Loki – Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and bolted-shipper, we also figured out how to configure alerts – Grafana Loki: alerts from the Loki Ruler and labels from logs. Now it’s time to figure out what we can do in Loki using its… Read More »
For general information on Grafana Loki, see the Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Among other services that make up Loki, there is a separate service called ruler that is responsible for working with alerts that can be generated directly from logs. The idea is very simple: create… Read More »
Prometheus has many ready-to-use exporters, but sometimes you may need to collect your own metrics. For this, Prometheus provides client libraries that we can use to generate metrics with the necessary labels. Such an exporter can be included directly in the code of your application, or it can be run as a separate service that will poll one… Read More »
The very useful thing is to monitor how efficiently the cluster is being used, especially if applications are deployed by developers who do not delve into requests much and set inflated values ”in reserve”. A reserve, of course, is needed – but simply requesting resources is a bad idea. For example, you have a WorkerNode… Read More »
The last time I’ve worked with the ELK stack about 7 years ago, see the ELK: установка Elasticsearch+Logstash+Kibana на CentOS. Currently, we are using Logz.io, but its costs going higher and higher, so we started looking at the self-hosted ELK solution to be running on our AWS Elastic Kubernetes Service clusters. So, the task, for… Read More »
Continuing with the theme of the AWS Trusted Advisor service, let’s see how can we configure notifications about new findings. The first part: AWS: Trusted Advisor, part 1 – its features overview. To have notifications, we can use Trusted Advisor metrics from the CloudWatch, see their list on the Trusted Advisor metrics and dimensions page.… Read More »
In the first post – AWS: Web Application Firewall overview, configuration, and its monitoring – we spoke about its main components, created a WebACL and Rules for it, and did basic monitoring. Also, we’ve configured WebACL’s logs collection with AWS Kinesis, but now it’s time to see them Logz.io, as CloudWatch Logs isn’t available for… Read More »
AWS CloudTrail is a service for auditing AWS accounts events and is enabled by default. It saves all actions that were done by a user, IAM role, or an AWS service via AWS Console, AWS CLI, or AWS SDK. CloudTrail will write information about every API call, log in to the system, services events, and… Read More »