Orchestrating Multiple AI Models:
Architecture Decisions

Sayan Dutta

Full-Stack Developer & AI Enthusiast · IIT Patna

Builder of the AI Integration Hub — a unified layer that orchestrates OpenAI, Anthropic, and Gemini behind a single API. This is the real architecture and the hard-won tradeoffs behind it.

When I first started integrating large language models into projects, I made the same mistake everyone makes: I picked one provider, wired it in directly, and shipped. Fast. Felt good. Then two months later I wanted to switch models for a specific task, and I realised my entire codebase was coupled to one vendor's SDK, response format, and error contract.

That's when I started building the AI Integration Hub — a routing layer that sits between my applications and every AI provider I use. This post is the technical breakdown of why it exists, how it's structured, and what I learned about the real tradeoffs between OpenAI, Anthropic, and Google Gemini that you won't find in any marketing page.

The Problem With Direct Integration

Every AI provider has its own SDK, its own request shape, its own error codes, its own rate-limiting behaviour, and its own way of streaming tokens. None of them agree on anything. Here's a simplified illustration of the same "summarise this text" task across three providers:

// OpenAI
const r1 = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: text }]
});
const out1 = r1.choices[0].message.content;

// Anthropic
const r2 = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  messages: [{ role: "user", content: text }]
});
const out2 = r2.content[0].text;

// Gemini
const r3 = await model.generateContent(text);
const out3 = r3.response.text();

Three different method names. Three different response shapes. Three different error types to catch. And that's before you add streaming, retry logic, or token counting. Every time you add a new provider or upgrade a model version, you touch application code. That's the problem the Hub solves.

The Architecture

The Hub is a Node.js service that exposes a single unified endpoint. All provider complexity lives inside it — the application never knows which model handled its request.

The request flow from any client is always the same: hit POST /complete with a task type, a prompt, and optional parameters. The Hub handles everything else — which provider to call, how to format the request, how to parse the response, and what to do when something fails.

The Routing Logic

The most important decision in a multi-model system isn't which models to support — it's how you decide which model handles a given request. Get this wrong and you're paying GPT-4o prices for tasks that a smaller model handles just as well, or routing safety-critical prompts to a model that hallucinates more freely.

Here's the decision logic I settled on after testing all three providers extensively:

// What every client receives, regardless of provider
{
  text:       "The model's response here",
  provider:   "anthropic",           // which provider served it
  model:      "claude-sonnet-4-6",   // exact model used
  tokens: {
    input:    142,
    output:   389,
    total:    531
  },
  latencyMs:  1240,
  cached:     false
}

The Real Tradeoffs: Provider by Provider

Every AI provider's marketing copy says the same things — fast, accurate, safe, cheap. Here's what I actually found after routing thousands of requests through all three:

The Caching Layer: More Impactful Than I Expected

I added Redis caching as an afterthought. It turned out to be the single highest-impact performance optimisation in the whole system.

The cache key is a hash of the normalised prompt + task type + model family (not exact model — so a model version bump doesn't invalidate everything). TTL varies by task type: factual Q&A caches for 24 hours, creative tasks for 1 hour, anything with a timestamp in the prompt doesn't cache at all.

function buildCacheKey(req) {
  const normalised = req.prompt
    .trim()
    .toLowerCase()
    .replace(/\s+/g, ' ');          // collapse whitespace

  const payload = JSON.stringify({
    p: normalised,
    t: req.task,
    f: modelFamily(req.task)      // "openai" | "anthropic" | "gemini"
  });

  return `hub:` + crypto
    .createHash('sha256')
    .update(payload)
    .digest('hex')
    .slice(0, 32);
}

const TTL = {
  factual_qa:   86400,   // 24h
  summarise:    3600,    // 1h
  code_review:  1800,    // 30min
  creative:     900,     // 15min
  safety_check: 0        // never cache
};

In practice, cache hit rates during Blix.ai beta testing — where classmates repeatedly ask similar CS theory questions — are around 38–45%. That's more than a third of all requests served from cache with sub-5ms latency and zero API cost.

Retry Logic and Graceful Degradation

Rate limit errors (429) and transient server errors (500, 503) are a fact of life with any AI provider. The Hub handles them with a three-tier strategy:

• Immediate retry — on a 429 with a Retry-After header, wait the specified duration and retry once on the same provider.
• Provider fallback — if the retry fails, or if there's no Retry-After, mark the provider degraded for 60 seconds and route to the fallback provider.
• Graceful error — if all providers fail (which has happened exactly twice across six months of testing), return a structured error with a retryable: true flag so the client can decide whether to surface it or queue for later.

What the Hub Does NOT Do

Scope creep is the enemy of reliable infrastructure. The Hub deliberately does not handle these things, despite being tempting to add:

• Prompt templating — that's the application's job. The Hub receives final prompts, not template variables.
• Context window management — the caller is responsible for fitting content within limits. The Hub will tell you if you've exceeded them, but it won't truncate on your behalf.
• Semantic caching — I prototyped embedding-based similarity matching to cache semantically equivalent prompts. The added complexity wasn't worth the marginal hit rate improvement. Exact-match hashing is good enough.
• Model evaluation — picking the right model for your specific task requires knowing your task. Hardcoded routing rules beat magic auto-selection for predictability.

"Make it work, make it right, make it fast — in that order." The Hub started as 80 lines of routing glue. The discipline of not adding features until they were clearly needed kept it maintainable as it grew.

Cost Monitoring in Practice

Every response logs { provider, model, inputTokens, outputTokens, latencyMs, taskType, cached } to a lightweight append-only log. A small cron job aggregates this daily into a cost summary. For the first month of running the Blix.ai beta through the Hub, the breakdown looked roughly like:

• 38% of all requests served from cache — zero API cost
• Of the remaining 62% that hit a provider:
  • 44% routed to Gemini — lowest per-token cost, used for summarisation tasks
  • 35% routed to Anthropic — complex CS theory questions needing precision
  • 21% routed to OpenAI — code-related questions and general Q&A

Without the Hub, every single request would have gone to one provider at full cost. With routing and caching, real-world spend during testing was roughly 55% lower than a naive single-provider setup would have cost.

Is a Multi-Model Architecture Right for You?

Honestly — not always. If you're building a quick prototype or a side project that calls one provider for one type of task, a routing layer adds complexity you don't need. Direct integration is fine.

You should consider building something like the Hub when:

• You're calling AI APIs in multiple projects and copying the same boilerplate
• You want to switch models without touching application code
• Your LLM spend is significant enough that routing optimisation has real ROI
• You're in a regulated domain where provider redundancy matters
• You want a single place to add features like caching, logging, and rate limit handling

The Hub took about three weeks to build properly. It's saved me far more than that in maintenance time and API costs since. The code is on my GitHub — feel free to fork it as a starting point.

Sayan Dutta

Full-Stack Developer & AI Enthusiast · IIT Patna

GitHub Get in touch