Multimodal RAG Production — Interactive Demo & Documentation

Interactive

Live in-browser demo

Type a recipe query below and watch the full RAG pipeline run in real time — embedding, retrieval, reranking, and generation. Everything executes client-side against 80 real recipes from the bundled sample dataset, using the same mock algorithms as the Python mock providers.

RAG Pipeline — Live

80 recipes indexed

Rerank Generate

Try: tomato basil pasta indian curry vegetarian chocolate dessert thai noodle soup fried chicken mexican tortilla

🧠

Stage 1

Embed

— ms

🔍

Stage 2

Retrieve

— ms

🎚️

Stage 3

Rerank

— ms

✨

Stage 4

Generate

— ms

📋 Retrieved & Reranked Recipes

🍽️

Run the pipeline to see results

✨ Generated Summary

Run the pipeline to generate a summary.

⏱️ Timing breakdown

embed	—
retrieve	—
rerank	—
generate	—
total	—

System Design

Architecture at a glance

Four stages, each behind a Protocol. Swap any layer without touching the others.

📝

Query

text, image, or both

→

🧠

Embedder

2048-dim vector

→

🗃️

Vector Store

Chroma / Qdrant / FAISS

→

🎚️

Reranker

optional, top-K re-scored

→

✨

Generator

natural-language summary

Every node is a Protocol with 3 implementations. Switch with one env var.

Step by Step

End-to-end expert walkthrough

From a fresh clone to a working RAG query in five steps. Each step shows the exact command and the expected output.

Install the package

~30 seconds · CPU-only · no GPU required

The default install pulls only the lightweight runtime dependencies. Mock providers boot instantly, so you can develop and test on any laptop.

bash

# Clone and install (mock providers by default)
git clone <repo> && cd multimodal-rag-production
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Optional: real NVIDIA Nemotron + Qwen3-VL models (needs ~16GB GPU)
# pip install -e ".[gpu]"

Configure providers via environment

.env file · pydantic-settings validates everything

Every knob is an environment variable. The defaults are safe for CPU; flip one variable to switch a provider.

bash

# .env — defaults shown
EMBEDDING_PROVIDER=mock        # mock | local_hf | openai
RERANKER_PROVIDER=mock         # mock | local_hf | openai
GENERATOR_PROVIDER=mock        # mock | local_hf | openai
VECTOR_STORE=chroma            # chroma | qdrant | faiss
DATASET_SOURCE=sample          # sample | hf | local
DEFAULT_TOP_K=5
RERANK_TOP_K=20
RATE_LIMIT_PER_MINUTE=60

Ingest the dataset into the vector store

~1 second for 200 sample recipes · ~10 minutes for full 10k HF dataset on GPU

The CLI embeds every recipe and upserts the vectors into the configured store. The sample dataset ships with the package; the full HuggingFace dataset streams on demand.

bash

# Ingest the bundled 200-recipe sample
multimodal-rag ingest --source sample

# Or pull the full 10,096-recipe HuggingFace dataset
# multimodal-rag ingest --source hf --limit 10096

# Expected output:
# Loading dataset source=sample limit=None
# Ingesting 200 recipes…
# Done. ingested=200 store=chroma

Behind the scenes, Embedder.embed_batch() turns each recipe into a 2048-dim L2-normalised vector, and VectorStore.upsert() writes the vector plus the recipe payload. For Chroma, nested metadata is flattened to top-level meta_* keys because Chroma only accepts scalar values.

Query the pipeline

CLI, REST API, or Python — same pipeline, three surfaces

The same RAGPipeline.run() method powers all three. The CLI is great for quick tests; the REST API is what production clients hit.

bash — CLI

multimodal-rag query \
  --text "tomato basil pasta" \
  --top-k 3

# Retrieved
#   1. Tomato Basil Pasta  (0.81)
#   2. Margherita Pizza    (0.74)
#   3. Caprese Salad       (0.71)
# Reranked
#   1. Tomato Basil Pasta  (0.92)
#   2. Margherita Pizza    (0.85)
# Summary
# Top pick: Tomato Basil Pasta…

bash — REST

curl -X POST http://localhost:8000/rag/query \
  -H "Content-Type: application/json" \
  -d '{
    "query_text": "tomato basil pasta",
    "top_k": 3,
    "rerank": true,
    "generate": true,
    "generate_style": "summary"
  }'

Inspect the response

JSON · per-stage timings · provider transparency

The response includes the retrieved items, the reranked items (if rerank was on), the generated summary, per-stage timings, and which providers handled each stage. This is exactly what the live demo above produces.

json — response

{
  "retrieved": [
    { "recipe": { "id": "recipe-0000", "title": "Tomato Basil Pasta", ... },
      "score": 0.81, "rank": 1 }
  ],
  "reranked": [
    { "recipe": { ... }, "score": 0.92, "rank": 1 }
  ],
  "summary": "Top pick: Tomato Basil Pasta — a classic Italian pasta…",
  "timings": {
    "retrieve": 0.012,
    "rerank":  0.034,
    "generate":0.156,
    "total":   0.202
  },
  "providers": {
    "embedding": "mock",
    "reranker":  "mock",
    "generator": "mock",
    "vector_store": "chroma"
  }
}

What you get

Production features

Every feature from the original notebook, plus the engineering scaffolding production systems need.

🔌

Pluggable providers

Embedder, Reranker, and Generator each have three implementations: mock local_hf openai. Switch with one env var.

🗃️

Three vector stores

chroma qdrant faiss behind one VectorStore Protocol. Embedded Chroma for dev, Qdrant for prod, FAISS for speed.

⚡

FastAPI REST API

Seven endpoints with auto OpenAPI docs: /health, /ingest, /retrieve, /rerank, /generate, /rag/query, /metrics.

🖥️

Typer CLI

serve, ingest, query, download-models, reset, info — Rich-formatted output, type-checked args.

🛡️

Production resilience

Retry with exponential backoff (tenacity), rate limiting (slowapi), in-memory LRU cache (cachetools), hierarchical exceptions mapped to HTTP codes.

📊

Observability built-in

Structured JSON logs (structlog) in prod, Prometheus metrics at /metrics, per-stage pipeline latency histograms, Grafana dashboard in docker-compose.

🐳

Docker + Compose

Multi-stage Dockerfile (CPU + GPU targets), 5-service docker-compose stack: API + Qdrant + Redis + Prometheus + Grafana. One command: docker compose up -d.

✅

Tested & CI-ready

55 pytest tests (54 passing, 1 skipped for Qdrant), 71% coverage, GitHub Actions CI on Python 3.10 / 3.11 / 3.12 with Docker smoke test.

🔒

Security-first

Optional API key (X-API-Key header), CORS allowlist, rate limiting per IP, image input validation, no secrets in logs.

📝

Type-checked throughout

Pydantic v2 schemas, mypy strict mode, full type hints on every public function. Protocol-typed providers mean no duck-typing surprises.

📦

Self-contained sample data

200-recipe JSON ships inside the package. multimodal-rag ingest --source sample works offline; no network needed.

📚

Documented

README, architecture / API / deployment / development / troubleshooting docs, plus a 30-section condensed DOCX technical documentation.

Under the hood

Technology stack

Conservative, well-maintained, permissively-licensed dependencies. Heavy ML libs are optional extras.

Layer	Library	Purpose
Web framework	FastAPI + Uvicorn	Async REST API + ASGI server
Validation	Pydantic v2 + pydantic-settings	Schema validation + env-var config
HTTP client	httpx + tenacity	HTTP calls with exponential-backoff retry
Rate limiting	slowapi	Per-IP sliding window rate limiter
Cache	cachetools	In-memory LRU cache decorator
Vector DB	Qdrant + ChromaDB + FAISS	Three interchangeable vector stores
ML (optional)	torch + transformers	NVIDIA Nemotron + Qwen3-VL inference
Datasets	datasets (HuggingFace)	Streaming the 10k recipe dataset
CLI	Typer + Rich	Typed CLI with pretty terminal output
Logging	structlog	Structured JSON logs in production
Monitoring	prometheus-client	Counters + histograms at `/metrics`
Testing	pytest + pytest-asyncio + pytest-cov	Unit + integration tests with coverage
Linting	ruff	Fast Python linter + formatter
Type checking	mypy	Static type analysis
Packaging	hatchling	Build backend for the wheel
Containers	Docker + docker-compose	Single-image + full-stack deployment
CI/CD	GitHub Actions	Lint, test, build matrix on push/PR

REST contract

API reference

Seven endpoints. Auto-generated OpenAPI at /docs (Swagger) and /redoc.

Endpoint	Method	Purpose	Auth
`/health`	GET	Liveness/readiness probe; returns provider + dataset info	—
`/ready`	GET	Kubernetes-style readiness probe	—
`/metrics`	GET	Prometheus exposition format	—
`/ingest`	POST	Load dataset (sample \| hf \| local) into vector store	✓
`/retrieve`	POST	Embed query and return top-K candidates	✓
`/rerank`	POST	Rerank supplied candidates against a query	✓
`/generate`	POST	Generate a summary over supplied recipes	✓
`/rag/query`	POST	End-to-end: retrieve → rerank → generate	✓

Example: end-to-end RAG query

bash — curl

curl -X POST http://localhost:8000/rag/query \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $API_KEY" \
  -d '{
    "query_text": "tomato basil pasta",
    "query_image": null,
    "top_k": 5,
    "rerank": true,
    "rerank_top_k": 20,
    "generate": true,
    "generate_style": "summary",
    "max_new_tokens": 512,
    "temperature": 0.7
  }'

Error responses

Every error returns a consistent JSON envelope. HTTP status is derived from the exception class.

HTTP	Class	When
400	`ValidationError`	Bad request payload
401	`AuthenticationError`	Missing/invalid `X-API-Key`
422	`DatasetError`	Dataset cannot be loaded
429	`RateLimitExceededError`	Rate limit hit
500	`RetrievalError` / `PipelineError`	Internal pipeline failure
502	`EmbeddingProviderError` etc.	Upstream provider failure
503	`ProviderNotAvailableError` / `VectorStoreConnectionError`	Service unavailable

Copy-paste ready

Code examples

The same pipeline, three ways: Python, CLI, and curl.

Python — use the pipeline directly

python

from multimodal_rag.pipeline import build_rag_pipeline

pipeline = build_rag_pipeline()

# Ingest the sample dataset
from multimodal_rag.data import build_dataset_loader
recipes = build_dataset_loader().load(source="sample")
pipeline.ingest(recipes)

# Run a full RAG query
result = pipeline.run(
    query_text="tomato basil pasta",
    top_k=3,
    rerank=True,
    generate=True,
    generate_style="summary",
)

print(result["summary"])
print(result["timings"])
# {'retrieve': 0.012, 'rerank': 0.034, 'generate': 0.156, 'total': 0.202}

Python — query with an image

python

from PIL import Image
from multimodal_rag.pipeline import build_rag_pipeline

pipeline = build_rag_pipeline()
img = Image.open("my_food_photo.jpg")

result = pipeline.run(
    query_text="what dish is this?",
    query_image=img,            # text + image fusion
    top_k=5,
    rerank=True,
    generate=True,
)
# The embedder averages the text and image embeddings (L2-normalised)
# so the combined vector stays in the same space as stored vectors.

CLI — full pipeline

bash

# Start the API
multimodal-rag serve --reload

# Ingest
multimodal-rag ingest --source sample

# Query
multimodal-rag query --text "tomato basil pasta" --top-k 3

# Switch to real models (one env var per provider)
EMBEDDING_PROVIDER=local_hf RERANKER_PROVIDER=local_hf GENERATOR_PROVIDER=local_hf \
  multimodal-rag serve

# Reset the vector store
multimodal-rag reset

Docker — full stack

bash

# Build + run the full stack: API + Qdrant + Redis + Prometheus + Grafana
docker compose up -d

# Tail logs
docker compose logs -f api

# API:   http://localhost:8000
# Qdrant UI: http://localhost:6333/dashboard
# Prometheus: http://localhost:9090
# Grafana:   http://localhost:3000  (admin / admin)

Retrieve recipes from text or images,rerank with a VLM, generate with an LLM.

Live in-browser demo

📋 Retrieved & Reranked Recipes

✨ Generated Summary

⏱️ Timing breakdown

Architecture at a glance

End-to-end expert walkthrough

Install the package

Configure providers via environment

Ingest the dataset into the vector store

Query the pipeline

Inspect the response

Production features

Pluggable providers

Three vector stores

FastAPI REST API

Typer CLI

Production resilience

Observability built-in

Docker + Compose

Tested & CI-ready

Security-first

Type-checked throughout

Self-contained sample data

Documented

Technology stack

API reference

Example: end-to-end RAG query

Error responses

Code examples

Python — use the pipeline directly

Python — query with an image

CLI — full pipeline

Docker — full stack

Retrieve recipes from text or images,
rerank with a VLM, generate with an LLM.