# Oscar LLM Tiering — World Class

## Problem
- 1.7B on VPS = 9-10s cold start, ~10s/reply warm. Acceptable but noticeable.
- 0.5B in browser = 398 MB download first time, then 0 cold-start forever after.

## Architecture: 3 tiers

| Tier | Where | Cost | Latency | When |
|---|---|---|---|---|
| **Tier 0** (in-browser, opt-in) | Phone/tablet browser cache | 398 MB one-time | <2s first reply, <500ms after | Greetings, short Q&A, cached standards, BS5837 quick lookup |
| **Tier 1** (VPS 1.7B, always available) | IONOS VPS via Cloudflare Tunnel | Network + 1.1 GB RAM VPS | 9-10s cold, ~10s warm | Blog rewrites, classification reasoning, anything Tier 0 can't do |
| **Tier 2** (VPS 1.7B, pre-warmed) | Same | Same | 0 cold (already loaded) | Once Tier 1 has been used once in the session |

## Hybrid routing

- Greeting detection: `prompt.length < 40` AND matches `^(hi|hello|hey|good (morning|afternoon|evening)|thanks|thank you|cheers)` → Tier 0
- Cached standards: `prompt.includes('BS5837')` AND `prompt.length < 200` → try cached first, fall through to Tier 0/1
- "Oscar, …" prefix or "Oscar, what is…" pattern → Tier 0
- Long / complex / "rewrite" / "draft" / "classify this oak" → Tier 1
- Tier 0 unavailable (not downloaded) → fall through to Tier 1

## Pre-warm strategy
- On app boot: fire a fire-and-forget `POST /api/chat` with `{prompt: "", num_predict: 1}` to Ollama
- This loads the model into RAM (KEEP_ALIVE=5m holds it)
- By the time the user types a real question, Tier 1 is hot
- The fire-and-forget request is a no-op from the user's perspective; the model loads silently

## Mobile RAM gating for Tier 0
- Check `navigator.deviceMemory` (if available — Chrome / Edge)
- 0.5B + KV cache = ~500-700 MB total in browser
- Only enable Tier 0 download button if `deviceMemory >= 4` GB (modern phones / tablets)
- Show clear "This downloads 398 MB" warning

## Phases

### Phase 1 (NOW): pre-warm + warm toast + hybrid routing
- Wire `ensureWarm()` that fires on app mount
- Add `useTierForPrompt(prompt)` to pick Tier 0 or 1
- Add "Oscar is loading — first reply takes ~10s" toast for first VPS call
- Streaming skeleton so user sees tokens appear even during cold-start

### Phase 2 (next): real 0.5B in-browser
- Bundle actual Qwen2.5-0.5B Q4_K_M GGUF as chunked assets in `public/brain/`
- Wire `@huggingface/transformers` for browser inference
- Add Settings → "Download offline brain" button with mobile gating
- Hybrid router uses 0.5B when available, falls through to VPS

### Phase 3 (later): cached standards lookup
- Embed the cached standards index in the bundle
- 0.5B can use a system prompt that points at cached entries
- Reduces Tier 0 cold replies to ~1-2s

## Why this is world class
- Instant for the 80% case (greetings + short Q&A → Tier 0)
- Sovereign when Tier 0 is offline (Tier 1 VPS, all on operator's own infra)
- Costs nothing extra at the network level (Tier 0 is local)
- Mobile-friendly (download is opt-in, gated on device capability)
- No third-party AI at any tier

---

## Phase 2 lessons (ambient AI, 2026-07-02)

When the cascade + pre-warm were working but the UX still felt like a
chat panel, the operator pushed: **"the input bar is always there and
if ai is aware of what page is open and what's been done on that page
they can answer and communicate through the input bar and toast system"**.

This is the right framing for any mobile AI surface. **Don't open a
new panel. Use the bar that's already there.**

### What we changed

- Deleted the entire chat-panel pattern (SheetMin / SheetFull /
  BackgroundNav / 4-state-machine).
- The input bar is now the chat. Three modes: idle, drafting, active.
- Active mode = pulsing status row with elapsed seconds inside the
  bar. No panel.
- Toasts at the top of the screen for completed work, errors, info.
- Apply pattern: AI never overwrites the user's text. Suggestions
  are drafts. User taps Apply to accept.

### What this transfers to

Any FVW app with an AI surface on mobile should use this pattern:

- FvRE pages with chat-on-page → input bar + toasts
- FreshVibe Studio sidebar assistant → input bar at the bottom
  of the canvas + toasts
- FreshCards quick-capture → input bar + toasts

The "AI is a separate surface" framing is a desktop affordance that
doesn't survive the mobile viewport. AI on mobile should be ambient
in the bar the user already has.

### New transferable rules

- The input bar is the chat. The toast is the response. The form
  is the canvas. Three surfaces, none of them modal.
- AI never overwrites user content. Always Apply, never auto-replace.
- Progress text reads faster than spinners. "Rewriting… 12s" beats
  a pulsing dot. Always have a status message.
- The × on a status row should say "stop waiting", not "stop the
  model" — the model keeps running in the background and the
  result is silently dropped if the user has moved on.

### Test

Phase 2: rewrite Notes in casual tone should:
- Show "Reading your text… → Looking up tone → Composing… → Writing…
  1243 tokens" in the bar with elapsed seconds
- The original Notes body stays in the editor throughout
- A toast appears at the top: "Rewrite in casual tone is ready ·
  1234 words · casual tone" with Preview / Apply / Keep original
- Apply replaces the body. Keep original dismisses. Preview shows
  the full text in an expandable section.
- During the rewrite, the user can type a follow-up question in
  the bar — it queues up and runs after the rewrite completes.