The next task I want to do with Golang is building a custom log collector that retrieves AWS Application Load Balancer logs from S3 and sends them to VictoriaLogs. Sure, we could simply use Vector.dev, like what I did for AWS VPC Flow Logs (see: Vector.dev introduction, logs from AWS S3, and VictoriaLogs integration). But… Read More »
Interfaces in Go allow you to describe access to data or behavior without providing concrete implementations inside the interface itself. In this way, we create a “common bus” that we can then use to “connect” external “systems.” In other words, an interface is an abstraction that defines a contract. The contract describes what can be… Read More »
Got a new task to monitor costs on OpenAI – to see how much each project spends per day, and send alerts to Slack if costs exceed a set threshold. I tried several existing exporters for the OpenAI API, but none provided cost metrics per project – so we’ll build one ourselves. To create the… Read More »
One day I received one of the default VictoriaMetrics alerts that are generated during the deployment of the Helm chart victoria-metrics-k8s-stack: I thought about writing a short post like “What is Churn Rate and how to fix it,” but in the end, I ended up diving deep into how VictoriaMetrics works with data in general… Read More »
We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes. The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10-11 gigabytes, which causes the Pod… Read More »
Let’s continue our journey with AWS OpenSearch Service. What we have is a small AWS OpenSearch Service cluster with three data nodes, used as a vector store for AWS Bedrock Knowledge Bases. Previous parts: AWS: Introduction to OpenSearch Service as a vector store AWS: Creating an OpenSearch Service cluster and configuring authentication and authorization Terraform:… Read More »
In the first part, we covered the basics of AWS OpenSearch Service in general and the types of instances for Data Nodes – AWS: Getting Started with OpenSearch Service as a Vector Store. In the second part, we covered access, AWS: Creating an OpenSearch Service Cluster and Configuring Authentication and Authorization. Now let’s write Terraform… Read More »
Ephemeral resources and write-only arguments appeared in Terraform a long time ago, back in version 1.10, but there was no opportunity to write about them in detail. The main idea behind them is not to leave “traces” in the state file, which is especially useful for passwords or tokens, because the data only exists during… Read More »
We are currently using AWS OpenSearch Service as a vector store for our RAG with AWS Bedrock Knowledge Base. We will talk more about RAG and Bedrock another time, but today let’s take a look at AWS OpenSearch Service. The task is to migrate our AWS OpenSearch Service Serverless to Managed, primarily due to (surprise)… Read More »
In the previous part, AWS: Getting Started with OpenSearch Service as a Vector Store, we looked at AWS OpenSearch Service in general, figured out how data is organized in it, what shards and nodes are, and what types of instances we actually need for data nodes. The next step is to create a cluster and… Read More »