AI & System Architecture

Deconstruction of asynchronous ingestion workers, vector database indexing, and streaming LLM pipelines.

RAG Ingestion Pipeline Stepper

7-stage pipeline deconstruction from S3 upload to streaming output

Multi-Format Ingestion

Amazon S3 / Presigned URLs

Accepts PDFs, text files, URLs, and YouTube video transcripts via presigned S3 upload URLs.

Architecture Mechanics:

Validates file size & MIME types

Issues secure temporary presigned uploads

Fires webhooks to trigger ingestion worker

S3 Ingestion Trigger Event:

{
  "fileId": "doc_92871",
  "fileName": "handbook_v2.pdf",
  "fileSize": "4.2 MB",
  "uploadUrl": "s3://jeem-ingest-bucket/uploads/handbook_v2.pdf",
  "mimeType": "application/pdf"
}

Stage 1 of 7

Complete Pipeline Architecture Specifications

Stage 01 — Multi-Format IngestionAmazon S3 Storage

Accepts PDFs, text files, URLs, and YouTube video transcripts via presigned S3 upload URLs.

• Validates file size & MIME types• Issues secure temporary presigned uploads• Fires webhooks to trigger ingestion worker

Stage 02 — Asynchronous Queue DispatchBullMQ & Redis / Valkey

Ingestion events are pushed into Redis task queues to process document parsing asynchronously.

• Prevents API timeouts during heavy document parsing• Automatic retry logic with exponential backoff• Concurrent worker scaling

Stage 03 — Document Extraction & ChunkingHybrid Extraction Engine

Parses text, removes formatting noise, and applies semantic boundary chunking with token overlaps.

• Sentence & paragraph boundary detection• Configurable chunk size (512 tokens + 64 overlap)• Preserves video timestamp metadata

Stage 04 — Dense Vector EmbeddingOpenAI Embeddings Engine

Generates high-dimensional vector representations (`text-embedding-3-small` / `large`) for each chunk.

• Batch embedding API requests• L2 normalized vectors for fast dot-product similarity• Includes payload metadata tags

Stage 05 — Vector Indexing & Hybrid SearchQdrant Vector Database

Stores embeddings in Qdrant collections with payload filtering for user/document scope isolation.

• Hybrid Dense + Sparse Keyword BM25 retrieval• Metadata payload filtering by document ID• Fast HNSW index searching

Stage 06 — Context Re-ranking & GuardrailsRe-ranker & Safety Gate

Re-ranks top-K retrieved candidates using cross-encoders and passes input through safety guardrails.

• Filters irrelevant chunks to reduce context bloat• Prevents prompt injection & toxicity• Computes evidence confidence scores

Stage 07 — LLM Synthesis & Evidence CitationStreaming Generation Engine

Synthesizes final answer with verbatim source citations, rendering right-sidebar evidence cards.

• Real-time SSE (Server-Sent Events) token streaming• Attaches source document page & timestamp badges• Renders right-sidebar evidence panel