Category Archives: Troubleshooting

Various problems solutions

LiteLLM: Debugging AI Cost Monitoring with VictoriaMetrics
5 (1)

16 September 2026

Had a pretty interesting case with monitoring LLM costs through LiteLLM when using multiple providers. What we have: LiteLLM: AI Gateway, all our services work through it, it proxies requests to providers, generates metrics and traces, controls access, spending, etc. OpenAI and OpenRouter: currently the two main providers LiteLLM sends requests to some clients (our… Read More: LiteLLM: Debugging AI Cost Monitoring with VictoriaMetrics5 (1) »

Loading

LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM
0 (0)

11 September 2026

A quick recap of what we are doing on the project right now: we have a separate hardware server (hostname == “Matrix”), where we run our own self-hosted models with llama.cpp. In our Kubernetes cluster we have a LiteLLM AI Gateway for our clients – Backend API and other project services. Clients send their OpenAI/Anthropic/OpenRouter… Read More: LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM0 (0) »

Loading

LiteLLM: Traffic Mirroring, Batch Completions, and Traffic to Two Providers
0 (0)

21 August 2026

We’re getting ready to launch a self-hosted LLM, and at the testing stage the general idea is to send client requests simultaneously both to the “default production model” like GPT-5.6 and to the model running on our own server. And after getting the responses, we’ll compare them with Phoenix or Opik, and gradually tune our… Read More: LiteLLM: Traffic Mirroring, Batch Completions, and Traffic to Two Providers0… »

Loading

LiteLLM: Metrics, Traces, and Debugging exception_class=”ValueError”
0 (0)

12 August 2026

A few days ago, I ran into an interesting situation with LiteLLM: on the one hand, the metrics showed a lot of errors “from the provider”, while on the other hand, the traces and alerts showed only a single error. I had to dig into it a bit and figure out some nuances of how… Read More: LiteLLM: Metrics, Traces, and Debugging exception_class=”ValueError”0 (0) »

Loading

NixOS: Getting Started, Installing Packages, and Configuring the System
0 (0)

5 August 2026

We got a new instance, a hardware server that will run our self-hosted LLMs. The server will run NixOS – not my choice, but the system looks interesting. I’ve been hearing about it for a long time, and now I have a great opportunity to get familiar with it. For now, my part is only… Read More: NixOS: Getting Started, Installing Packages, and Configuring the System0 (0) »

Loading

VictoriaTraces: Recording Rules, Metrics, and Alerts from Trace Spans
0 (0)

22 May 2026

VictoriaTraces – just like VictoriaLogs – supports Recording Rules (see VictoriaMetrics: Recording Rules for AWS Load Balancer logs) for traces, because traces are essentially the same logs, just structured differently. And since we have Recording Rules – we can build metrics out of logs for alerts and Grafana dashboards. Although this actually isn’t the best… Read More: VictoriaTraces: Recording Rules, Metrics, and Alerts from Trace Spans0 (0) »

Loading

Arch Linux: a DNS Mystery – VPN, systemd-resolved, and Unbound
0 (0)

21 May 2026

I’d been wrestling with the problem of accessing AWS EKS from the office for a long time – finally lost my patience and figured it out 🙂 Here’s the problem: there’s an AWS EKS cluster with both Public and Private endpoints for the API. Working from my office laptop, sometimes requests to it go through… Read More: Arch Linux: a DNS Mystery – VPN, systemd-resolved, and Unbound0… »

Loading

OpenTelemetry: OTel Collectors in Kubernetes and VictoriaMetrics Stack integration
0 (0)

16 May 2026

Today let’s talk about how to run OpenTelemetry in Kubernetes and integrate it with the VictoriaMetrics stack – VictoriaMetrics for metrics, VictoriaLogs for logs, and VictoriaTraces for traces. Actually, this post wasn’t planned at all, and once a draft did appear – it was supposed to be the third in the series, but in the… Read More: OpenTelemetry: OTel Collectors in Kubernetes and VictoriaMetrics Stack integration0 (0) »

Loading

Arch Linux: WireGuard Peer for Connecting to MikroTik
0 (0)

8 May 2026

In the post MikroTik: setting up WireGuard and connecting Linux peers I described how to set up MikroTik as a VPN Hub and connect a peer running on Debian Linux. Setup on Arch Linux is mostly the same as on Debian – but every time I end up digging through this blog and my other… Read More: Arch Linux: WireGuard Peer for Connecting to MikroTik0 (0) »

Loading

SSL/TLS: Self-signed Certificate Authority for NGINX on FreeBSD
0 (0)

18 April 2026

I run a bunch of web services on my home NAS – Grafana, VictoriaMetrics, my own WordPress blog, and half a dozen other small things. The whole series of posts on FreeBSD and NAS starts here – FreeBSD: Home NAS, part 1 – setting up ZFS mirror, there are 15 parts as of now. NGINX+PHP… Read More: SSL/TLS: Self-signed Certificate Authority for NGINX on FreeBSD0 (0) »

Loading