This one turned out to be a bit unusual 🙂 Unusual in the sense that it’s actually quite “usual” – usual in the way posts like this were written back in the old days, in the age of mammoths – before LLMs and AI agents: today I was debugging an issue, and decided to do… Read More: LiteLLM: PostgreSQL Failed Requests, AWS RDS, and “Manual Debugging”0 (0) »
A very random text. Wasn’t planned at all, and there aren’t that many posts like this on this blog, but sometimes something “clicks in my head” (while there’s still something there to click) – and a thought takes shape that I want to express in a text like this, to save it. “Agentic development is… Read More: AI: LLMs, Agents, Work, and Us – Engineers. Personal Thoughts.0… »
The main goal we had when introducing LiteLLM AI Gateway on the project (see LiteLLM: AI Gateway for LLMs – features overview) was access and cost control, because right now we can’t enable auto-recharge for our OpenAI or OpenRouter account: our startup is still in the development and experimentation stage, developers actively use AI agents… Read More: AI: LLM Spend Control with LiteLLM Budgets and OpenAI Limits0… »
Had a pretty interesting case with monitoring LLM costs through LiteLLM when using multiple providers. What we have: LiteLLM: AI Gateway, all our services work through it, it proxies requests to providers, generates metrics and traces, controls access, spending, etc. OpenAI and OpenRouter: currently the two main providers LiteLLM sends requests to some clients (our… Read More: LiteLLM: Debugging AI Cost Monitoring with VictoriaMetrics5 (1) »
A quick recap of what we are doing on the project right now: we have a separate hardware server (hostname == “Matrix”), where we run our own self-hosted models with llama.cpp. In our Kubernetes cluster we have a LiteLLM AI Gateway for our clients – Backend API and other project services. Clients send their OpenAI/Anthropic/OpenRouter… Read More: LiteLLM: Custom Callbacks and LLM Evaluations with Judge LLM0 (0) »
We’re currently setting up our own hardware server where we want to run self-hosted models. But we can’t just switch client traffic to them right away – first we need to see how these self-hosted LLMs will actually perform. So the general idea for now is to keep sending traffic to the primary provider, OpenAI… Read More: LiteLLM: Custom Callback for Traffic Mirroring and OTel Tracing to… »
We’re getting ready to launch a self-hosted LLM, and at the testing stage the general idea is to send client requests simultaneously both to the “default production model” like GPT-5.6 and to the model running on our own server. And after getting the responses, we’ll compare them with Phoenix or Opik, and gradually tune our… Read More: LiteLLM: Traffic Mirroring, Batch Completions, and Traffic to Two Providers0… »
We have a server where we’re going to run self-hosted LLMs. We spent quite a while choosing what exactly to use for running the models – vLLM, SGLang, or llama.cpp, and eventually settled on llama.cpp – at least for now. In the post NixOS: getting started, package installation, and system configuration I described installing Node… Read More: llama.cpp: Metrics and Monitoring with VictoriaMetrics0 (0) »
We have an Ubuntu server running one of our services. The service itself is still at the Proof of Concept stage, so we’re not moving it to Kubernetes yet – but sending traffic in plain text, especially when these are requests to AI Agents with prompts and responses, is not a great idea. So, the… Read More: Ubuntu: Installing NGINX with TLS Certificate from Let’s Encrypt and… »
A few days ago, I ran into an interesting situation with LiteLLM: on the one hand, the metrics showed a lot of errors “from the provider”, while on the other hand, the traces and alerts showed only a single error. I had to dig into it a bit and figure out some nuances of how… Read More: LiteLLM: Metrics, Traces, and Debugging exception_class=”ValueError”0 (0) »