Category Archives: HOWTO’s

LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM
0 (0)

11 September 2026

A quick recap of what we are doing on the project right now: we have a separate hardware server (hostname == “Matrix”), where we run our own self-hosted models with llama.cpp. In our Kubernetes cluster we have a LiteLLM AI Gateway for our clients – Backend API and other project services. Clients send their OpenAI/Anthropic/OpenRouter… Read More: LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM0 (0) »

Loading

LiteLLM: Custom Callback for Traffic Mirroring and OTel Tracing to VictoriaTraces
0 (0)

9 September 2026

We’re currently setting up our own hardware server where we want to run self-hosted models. But we can’t just switch client traffic to them right away – first we need to see how these self-hosted LLMs will actually perform. So the general idea for now is to keep sending traffic to the primary provider, OpenAI… Read More: LiteLLM: Custom Callback for Traffic Mirroring and OTel Tracing to… »

Loading

llama.cpp: Metrics and Monitoring with VictoriaMetrics
0 (0)

21 August 2026

We have a server where we’re going to run self-hosted LLMs. We spent quite a while choosing what exactly to use for running the models – vLLM, SGLang, or llama.cpp, and eventually settled on llama.cpp – at least for now. In the post NixOS: getting started, package installation, and system configuration I described installing Node… Read More: llama.cpp: Metrics and Monitoring with VictoriaMetrics0 (0) »

Loading

Ubuntu: Installing NGINX with TLS Certificate from Let’s Encrypt and AWS Route 53
0 (0)

17 August 2026

We have an Ubuntu server running one of our services. The service itself is still at the Proof of Concept stage, so we’re not moving it to Kubernetes yet – but sending traffic in plain text, especially when these are requests to AI Agents with prompts and responses, is not a great idea. So, the… Read More: Ubuntu: Installing NGINX with TLS Certificate from Let’s Encrypt and… »

Loading

LiteLLM: Metrics, Traces, and Debugging exception_class=”ValueError”
0 (0)

12 August 2026

A few days ago, I ran into an interesting situation with LiteLLM: on the one hand, the metrics showed a lot of errors “from the provider”, while on the other hand, the traces and alerts showed only a single error. I had to dig into it a bit and figure out some nuances of how… Read More: LiteLLM: Metrics, Traces, and Debugging exception_class=”ValueError”0 (0) »

Loading

NixOS: Getting Started, Installing Packages, and Configuring the System
0 (0)

5 August 2026

We got a new instance, a hardware server that will run our self-hosted LLMs. The server will run NixOS – not my choice, but the system looks interesting. I’ve been hearing about it for a long time, and now I have a great opportunity to get familiar with it. For now, my part is only… Read More: NixOS: Getting Started, Installing Packages, and Configuring the System0 (0) »

Loading

LiteLLM: OpenRouter Integration and Fallbacks Configuration
0 (0)

31 July 2026

OpenRouter recently announced a 50% discount on OpenAI, so we decided to give it a try. We’ll switch things over on LiteLLM, which already handles requests from all our services – so the switch should be pretty simple. Before OpenRouter, we’ll set up fallbacks straight to OpenAI in LiteLLM – for cases when requests to… Read More: LiteLLM: OpenRouter Integration and Fallbacks Configuration0 (0) »

Loading

MongoDB: run in Kubernetes with MongoDB Operator and MongoDBCommunity CRD
0 (0)

24 July 2026

Following up on the previous post Valkey: running in Kubernetes – Helm, monitoring, ACL – we set up Valkey/Redis there, now it’s time to add MongoDB. This is for our new service, which is still in the experimental/PoC phase, but will most likely go to production, so the setup is a bit “under-production” – we’re… Read More: MongoDB: run in Kubernetes with MongoDB Operator and MongoDBCommunity CRD0… »

Loading

Valkey: Deployment in Kubernetes – Helm, Monitoring, ACL
0 (0)

22 July 2026

We’re launching a new service for the project, and this service needs Redis and MongoDB. The “technical task” itself looks roughly like this: Redis: task queue + some other stuff (e.g. we stream the agent response from LLM to Redis, and if the user is connected on a websocket – then we stream data from… Read More: Valkey: Deployment in Kubernetes – Helm, Monitoring, ACL0 (0) »

Loading

LiteLLM: Monitoring with VictoriaMetrics – Alerts and Grafana
0 (0)

17 July 2026

This is the second part of the LiteLLM monitoring series – in the previous one, we covered the general integration with VictoriaStack and looked at the metrics and traces we get from LiteLLM (see LiteLLM: metrics, traces, and integration with the VictoriaMetrics Stack). Now let’s move on to the practical part – what to monitor… Read More: LiteLLM: Monitoring with VictoriaMetrics – Alerts and Grafana0 (0) »

Loading