CONDUITWHY I BUILT CONDUIT
Every LLM-powered product eventually hits the same wall: API costs scale linearly with usage, and most requests don't actually need a frontier model to answer them. I wanted to build the thing that solves this properly, not a wrapper around one API, but a real inference gateway that decides, per request, which model is worth paying for. The constraint I set was deliberate: ₹0 budget, full stop. No paid APIs, no credit card, nothing. That constraint made the project harder in the right ways, free-tier providers have real, tight rate limits, so failover and quota-aware routing stopped being a "nice to have" and became a survival requirement. The system had to actually hold up under pressure, not just work in a demo. Conduit classifies every request for difficulty, serves repeat questions instantly from a semantic cache, and automatically fails over across four providers, down to a self-hosted local model that never goes offline, the moment anything upstream breaks. Every decision it makes is logged and explained, not hidden behind a black box. None of this was built in a straight line. It started with research spikes before a line of code existed, went through formal architecture decisions along the way, and, more than once, real debugging when something that looked fine turned out not to be. That process is part of the project, not something to hide.
Architecture
How Conduit Is Built
Key Engineering Decisions
Learned Routing, Not LLM-as-Router
A trained classifier decides difficulty in milliseconds on CPU. Asking another LLM to route would add latency and cost to every single request.
pgvector Over a Dedicated Vector DB
One less system to run and monitor. At this scale, a dedicated vector database is solving a problem that doesn't exist yet.
Free-Tier Multi-Provider Strategy
No single provider is reliable enough alone. The failover chain isn't a demo feature, it's survival.
Self-Hosted Local Model as the Final Tier
The one tier that never runs out of quota. Everything else can fail; this can't.
About the Developer
Want to Know More?
For the full technical breakdown, including request lifecycle, API contract, database schema, and every architecture decision explained, read the complete architecture document.