Groq Dual-AI Integration: Why I Added a Second AI and What It Actually Fixed

I added a second AI to VORA. Not because one wasn't enough. Because one was too slow for one specific task, and that one task happened to be the one users notice most.

Then the second AI broke on day one with a 400 error. Because of course it did.

⏱️ The Latency Problem I Couldn't Ignore

After solving the 429 rate limit chaos (that's a whole other post about priority queues), VORA's Gemini integration was stable. But stability revealed something I'd been ignoring: latency asymmetry.

VORA has three types of AI tasks, and they don't all need the same speed:

• CORRECTION: Needs to feel instant -- 1-2 seconds max. Gemini Flash hits this about 70% of the time. The other 30%? 3-5 seconds. In real-time transcription, a five-second delay is an eternity. It's like watching someone type a text message one letter at a time.
• QA: Users can wait 2-3 seconds for a question answer. No rush.
• SUMMARY: Background task. Could take a minute and nobody would care.

Gemini Flash is a great generalist. But for the quick, repetitive correction task, I needed something faster. Groq's Llama 3.3 70B, running on their custom LPU hardware, delivers sub-1-second responses for short prompts. That's the difference between "the correction appeared" and "the correction was already there."

🏗️ The Architecture: Opt-In, Not Forced

I didn't want to make everyone manage two API keys. Most users are fine with Gemini-only.

So I built it as a toggle: "Fast Correction Mode" in settings. Turn it on, CORRECTION tasks route to Groq. QA and SUMMARY stay on Gemini. Turn it off (or don't configure Groq), everything stays on Gemini like before.

It's like having a sous-chef for the quick prep work while the head chef handles the complex dishes. The sous-chef (Groq) is fast and precise for the repetitive stuff. The head chef (Gemini) takes more time but handles the nuanced, contextual work. Both in the kitchen. Neither confused about their job.

The TextCorrector module already had an aiCorrect() method. I just added a GrokAPI class that matched the Gemini interface, wired up the toggle, and routed based on the useGrokForCorrection flag. Clean enough.

🚨 Day One: The 400 Error

Deployed the integration. First real test session. Every single request: HTTP 400 from api.groq.com/openai/v1/chat/completions. Just "Bad Request." No helpful error message. Nothing.

The bug was embarrassing:

// The bug:
this.model = 'llama-3.3-70b';           // Does not exist as a Groq model ID
 
// The fix:
this.model = 'llama-3.3-70b-versatile'; // Correct Groq model identifier (128K ctx)

I typed the model name from memory instead of copying it from Groq's console. The API requires the full identifier including the deployment variant -- -versatile for the general-purpose 128K-context deployment. Without it, the name doesn't resolve to anything. It's like ordering "the chicken" at a restaurant with five chicken dishes. Close doesn't count when you're talking to an API.

Lesson I'll forget and relearn in three months: always copy-paste identifiers from the provider's actual dashboard. Never from your brain.

The silver lining: while debugging, I discovered Groq's API follows the OpenAI-compatible format. temperature, max_tokens, standard message roles -- all work exactly the same. Made the rest of the integration much simpler than expected.

✅ The Fallback Pattern

Even after fixing the model name, I needed to handle the inevitable: what if Groq goes down? Rate limit? Network hiccup? I can't let the correction pipeline silently break.

async _grokCorrect(text) {
    try {
        const result = await this.grokAPI.correctText(text, prompt);
        if (result) { /* process and return */ }
        return { text: text, isQuestion: false }; // empty result fallback
    } catch (error) {
        console.warn('[Groq] Correction failed, falling back to Gemini:', error.message);
        if (this.geminiAPI && this.geminiAPI.isConfigured) {
            return this._geminiCorrectOriginal(text);
        }
        return { text: text, isQuestion: false }; // worst case: no correction
    }
}

Fallback chain: Groq → Gemini (if configured) → original text unchanged. Users might get slightly less correction if Groq is down, but they never see a broken state. Worst case: the text stays as-is. That's infinitely better than a crash or a hang.

📊 The Numbers

With Groq handling corrections and Gemini handling the deeper tasks:

• Groq correction latency: median ~650ms, 95th percentile ~1.2s
• Gemini correction latency: median ~900ms, 95th percentile ~2.8s

That 250ms median difference sounds small on paper. In practice, when corrections are happening continuously during a live meeting, it's the difference between the app feeling responsive and the app feeling laggy.

Bonus: Gemini's priority queue is way less congested now. Correction tasks used to be the majority of queue items. Now they're routed to Groq. QA answers come back faster because they're not waiting behind a wall of correction requests.

📈 The Admin Dashboard

I extended the admin monitoring page to track both AI systems. The heartbeat broadcasts via BroadcastChannel from the app tab, now including Groq stats: grokEnabled, grokStats.totalRequests, grokStats.totalErrors, grokStats.avgLatency.

Open it in a separate tab during a long meeting session, and you can watch both AI systems in real time. Originally built for my own debugging, but it turns out to be genuinely useful for investigating user-reported issues too.

⚠️ Where This Falls Short

I'll be honest: for most users running a free Gemini API key, the Groq upgrade probably isn't worth the friction of getting a second API key. The difference between 900ms and 650ms is real but subtle. Most people won't notice.

Where it does matter: power users in multi-hour meetings. Over two hours, the compounding effect of faster corrections adds up.

There's also a rate-limiting gap I haven't solved yet. Groq's free tier allows ~30 requests/min. If Groq gets rate-limited, it falls back to Gemini -- which means Gemini suddenly absorbs correction tasks it wasn't budgeted for. A smarter version would track combined correction rates and pick the model with more headroom. That's a future fix.

🧠 What I Learned

Running two AI models only makes sense when they have genuinely different strengths mapped to genuinely different tasks. Groq is fast for short prompts. Gemini is deep for long context. That's a real division of labor.

If you're just using two models as hot-standbys for each other, save yourself the complexity. Use one model with good retry logic instead.

2026.02.12