Category Archives: AI

LiteLLM: Debugging AI Cost Monitoring with VictoriaMetrics
0 (0)

16 September 2026

Had a pretty interesting case with monitoring LLM costs through LiteLLM when using multiple providers. What we have: LiteLLM: AI Gateway, all our services work through it, it proxies requests to providers, generates metrics and traces, controls access, spending, etc. OpenAI and OpenRouter: currently the two main providers LiteLLM sends requests to some clients (our… Read More: LiteLLM: Debugging AI Cost Monitoring with VictoriaMetrics0 (0) »

Loading

LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM
0 (0)

11 September 2026

A quick recap of what we are doing on the project right now: we have a separate hardware server (hostname == “Matrix”), where we run our own self-hosted models with llama.cpp. In our Kubernetes cluster we have a LiteLLM AI Gateway for our clients – Backend API and other project services. Clients send their OpenAI/Anthropic/OpenRouter… Read More: LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM0 (0) »

Loading

LiteLLM: Custom Callback for Traffic Mirroring and OTel Tracing to VictoriaTraces
0 (0)

9 September 2026

We’re currently setting up our own hardware server where we want to run self-hosted models. But we can’t just switch client traffic to them right away – first we need to see how these self-hosted LLMs will actually perform. So the general idea for now is to keep sending traffic to the primary provider, OpenAI… Read More: LiteLLM: Custom Callback for Traffic Mirroring and OTel Tracing to… »

Loading

LiteLLM: Traffic Mirroring, Batch Completions, and Traffic to Two Providers
0 (0)

21 August 2026

We’re getting ready to launch a self-hosted LLM, and at the testing stage the general idea is to send client requests simultaneously both to the “default production model” like GPT-5.6 and to the model running on our own server. And after getting the responses, we’ll compare them with Phoenix or Opik, and gradually tune our… Read More: LiteLLM: Traffic Mirroring, Batch Completions, and Traffic to Two Providers0… »

Loading

llama.cpp: Metrics and Monitoring with VictoriaMetrics
0 (0)

21 August 2026

We have a server where we’re going to run self-hosted LLMs. We spent quite a while choosing what exactly to use for running the models – vLLM, SGLang, or llama.cpp, and eventually settled on llama.cpp – at least for now. In the post NixOS: getting started, package installation, and system configuration I described installing Node… Read More: llama.cpp: Metrics and Monitoring with VictoriaMetrics0 (0) »

Loading

LiteLLM: Metrics, Traces, and Debugging exception_class=”ValueError”
0 (0)

12 August 2026

A few days ago, I ran into an interesting situation with LiteLLM: on the one hand, the metrics showed a lot of errors “from the provider”, while on the other hand, the traces and alerts showed only a single error. I had to dig into it a bit and figure out some nuances of how… Read More: LiteLLM: Metrics, Traces, and Debugging exception_class=”ValueError”0 (0) »

Loading

LiteLLM: OpenRouter Integration and Fallbacks Configuration
0 (0)

31 July 2026

OpenRouter recently announced a 50% discount on OpenAI, so we decided to give it a try. We’ll switch things over on LiteLLM, which already handles requests from all our services – so the switch should be pretty simple. Before OpenRouter, we’ll set up fallbacks straight to OpenAI in LiteLLM – for cases when requests to… Read More: LiteLLM: OpenRouter Integration and Fallbacks Configuration0 (0) »

Loading

LiteLLM: Monitoring with VictoriaMetrics – Alerts and Grafana
0 (0)

17 July 2026

This is the second part of the LiteLLM monitoring series – in the previous one, we covered the general integration with VictoriaStack and looked at the metrics and traces we get from LiteLLM (see LiteLLM: metrics, traces, and integration with the VictoriaMetrics Stack). Now let’s move on to the practical part – what to monitor… Read More: LiteLLM: Monitoring with VictoriaMetrics – Alerts and Grafana0 (0) »

Loading

LiteLLM: Metrics, Traces, and VictoriaMetrics Stack Integration
0 (0)

9 July 2026

Third part on running LiteLLM – AI Gateway or LLM Proxy, and finally we’re getting to monitoring. In the first part we got familiar with LiteLLM in general (see LiteLLM: AI Gateway for LLMs – overview of features), and in the second one we deployed it in Kubernetes and hooked up VictoriaTraces and VictoriaMetrics to… Read More: LiteLLM: Metrics, Traces, and VictoriaMetrics Stack Integration0 (0) »

Loading

LiteLLM: AI Gateway on Kubernetes and Metrics in VictoriaMetrics
0 (0)

26 June 2026

In the first part – LiteLLM: AI Gateway for LLMs – features overview we got familiar with what LiteLLM can do in general – now we can run it in Kubernetes and connect clients. At the same time we’ll check the integration with our existing monitoring stack – for now just metrics to VictoriaMetrics. Logs… Read More: LiteLLM: AI Gateway on Kubernetes and Metrics in VictoriaMetrics0 (0) »

Loading