How to Build a Custom AI Chatbot for Your Business Without OpenAI Lock-In

Every second BFSI client who walks into Speqto with a chatbot requirement asks the same question halfway through the first call: “What happens if OpenAI changes pricing, or our regulator flags us for routing customer data to a US-based model?” It’s a fair worry. We’ve seen NBFCs and payment aggregators build quick chatbot MVPs on the OpenAI API, only to hit a wall six months later when compliance teams ask where the data actually goes, or when a pricing change quietly triples the monthly bill.

This post is about the architecture we actually use to avoid that trap — not a theoretical “best practices” list, but the real decisions we made while building chatbots for an NBFC lending platform and a payments company’s support desk.

Why lock-in actually hurts BFSI companies

It’s not just about switching costs. For regulated businesses, the real risks are:

• Data residency: RBI and most state regulators expect customer financial data to stay within defined boundaries. Sending raw loan application text or KYC details to a third-party API outside your control is a compliance headache, even if the vendor claims “no training on your data.”
• Pricing volatility: One of our clients, a mid-sized NBFC, saw their token costs jump nearly 40% within a quarter after a provider updated rate cards. Their chatbot volume hadn’t even grown that much — the model’s context window usage had just increased because of longer conversation threads.
• Model deprecation: Providers retire model versions on their own schedule. If your prompts and fine-tuning are tightly coupled to one specific model, an upgrade notice becomes an emergency sprint.
• Single point of failure: If the provider has an outage (and every major LLM API has had one), your customer support chatbot goes down with it.

The fix: build an abstraction layer, not a chatbot tied to one API

The core idea is simple — your application should never call “OpenAI’s API” directly. It should call your own internal chat service, which then decides which model to route the request to. This one architectural decision is what separates a chatbot you own from a chatbot you’re renting.

Here’s roughly how we structured it for a fintech client building a loan-servicing assistant:

• Orchestration layer: We used LiteLLM as a routing proxy, which exposes an OpenAI-compatible endpoint but can forward requests to Azure OpenAI, Anthropic, or a self-hosted Llama 3 model depending on rules we define — query type, data sensitivity, or even current cost thresholds.
• Self-hosted model for sensitive queries: Anything touching PII — loan amount, PAN details, repayment status — gets routed to a Llama 3 70B model running on a dedicated GPU instance inside their own VPC. Nothing leaves their infrastructure.
• Commercial API for generic queries: FAQ-type questions (“What documents do I need for a personal loan?”) get routed to a cheaper, faster commercial model since there’s no sensitive data involved. This alone cut their monthly inference cost by about a third.
• RAG on their own documents: We used Qdrant as a self-hosted vector database instead of a managed one, storing embeddings of their product policies, loan terms, and support scripts. This means the knowledge base — the actual IP — sits in infrastructure they control, not inside a vendor’s dashboard.

Practical steps if you’re starting from scratch

1. Map your data sensitivity first, not your model choice. Decide which conversation types involve regulated data before picking any vendor. This single exercise usually decides 80% of your architecture.
2. Pick an orchestration layer early. Tools like LiteLLM or a simple internal API gateway let you swap models without rewriting your chatbot logic. Treat the model as a config value, not a hardcoded dependency.
3. Test with at least two model providers from day one. Even if you plan to launch with one, build and test your prompts against a second (say, Mistral or Llama 3 via Ollama/vLLM) so you know your prompt engineering isn’t accidentally tuned to one vendor’s quirks.
4. Keep your embeddings and vector store self-managed. Your knowledge base is the real asset here — the model is replaceable, your data pipeline shouldn’t be.
5. Build a fallback path. When we built the payments company’s support bot, we added automatic failover: if the primary model API times out, the request reroutes to the self-hosted model within seconds, so customers never see a blank chat window.

What this costs in practice

Running a self-hosted Llama 3 model isn’t free — GPU instances aren’t cheap, and you need someone to maintain uptime. For the NBFC client, the break-even came around month four, once their monthly conversation volume crossed roughly 50,000 sessions. Below that volume, a hybrid setup (mostly commercial API, self-hosted only for sensitive flows) made more financial sense than going fully self-hosted.

Where Speqto fits in

We don’t push clients toward self-hosting everything — that’s its own kind of lock-in, just with more maintenance overhead. What we do push for is optionality: an architecture where switching models, adding a new provider, or moving a workload in-house for compliance reasons takes a config change, not a rebuild. If you’re a BFSI or fintech team evaluating chatbot vendors right now, the question to ask isn’t “which model do you use” — it’s “can I change that answer later without starting over.”