Problem

Users need to ask questions of their own PDF documents and get answers traceable back to the source pages, with per-user document isolation.

What it does

A monorepo app where users sign in via Clerk, upload PDFs from a dashboard, and chat with them. A BullMQ worker extracts PDF text, chunks it (1500/200), tags chunks with docId/userId metadata, and upserts them into a Qdrant 'docs' collection; chat requests run a LangChain v1 createAgent loop whose retrieveContext tool does metadata-filtered Qdrant search, and the LLM returns a structured {result, references} payload with inline [1]-style page citations. Conversations, chats, and docs are persisted in Postgres via Drizzle; a Redis cache and a LangGraph Redis checkpointer hold per-conversation state.

Architecture

[frontend] Next.js 16 frontend (landing, dashboard, chat client) (client)
[api] Express 5 API (Clerk auth, doc/chat routes) (api)
[worker] BullMQ worker (PDF ingestion + chat persistence) (worker)
[postgres] Supabase Postgres (Drizzle: users/docs/conversations/chats) (store)
[redis] Redis (queues, conversation cache, LangGraph checkpointer) (store)
[qdrant] Qdrant collection 'docs' (vector store) (store)
[objectstorage] Oracle Cloud Object Storage (uploaded PDFs) (external)
[clerk] Clerk auth + svix webhooks (external)
[llm] DeepSeek chat LLM via ChatDeepSeek (model)
[embeddings] Gemini gemini-embedding-001 embeddings (model)

frontend ──▶ api · axios/fetch with Clerk Bearer token, chunked JSON stream parsing
api ──▶ clerk · clerkMiddleware + getAuth per route
clerk ──▶ api · svix-verified user.created/updated/deleted webhook
api ──▶ worker · enqueue file-processing jobs; upload blocks on job.waitUntilFinished (120s)
worker ──▶ postgres · insert docs row; persist Q/A pairs into chats
worker ──▶ qdrant · upsert chunk embeddings tagged docId/userId
worker ──▶ objectstorage · upload original PDF, store public URL
api ──▶ llm · agent.streamEvents with retrieveContext tool; LLM titles new conversations
api ──▶ qdrant · top-5 similarity search filtered by metadata.docId + userId; delete vectors on doc delete
api ──▶ redis · last-10-responses cache + RedisSaver thread checkpointer (TTL 3600s)
api ──▶ postgres · doc ownership checks, conversations/chats reads
api ──▶ embeddings · embed search query for Qdrant retrieval

Simplified — box-and-arrow sketch of the repo structure, not a system diagram.

Decisions

1. 01

   PDF ingestion runs as a BullMQ job the API blocks on

   createDoc enqueues a file-processing job then awaits job.waitUntilFinished(events, 120_000), so the HTTP response can return the created doc row (with page count) while keeping the heavy parsing out of the request cycle — though it serializes uploads rather than returning immediately.

   Evidence: apps/backend/src/controllers/doc.controller.ts

2. 02

   Agentic RAG with a metadata-filtered retrieve tool instead of pre-retrieved context

   The LangGraph agent decides when to call retrieveContext, whose Qdrant query filters on metadata.docId AND metadata.userId from the runtime contextSchema, enforcing multi-tenant isolation at retrieval time rather than trusting the model.

   Evidence: apps/backend/src/config/ai.ts

3. 03

   Structured, page-cited answers plus prompt-injection hardening

   responseFormat uses toolStrategy on a {result, references} schema; the system prompt demands inline [n] citations mapped to a references array, forbids fabricated page numbers, and instructs the model to treat retrieved PDF text as untrusted data, not instructions.

   Evidence: apps/backend/src/config/ai.ts

4. 04

   Client-side parser for delimiter-less chunked JSON

   The backend streams back-to-back JSON objects with no delimiter and never ends the response, so the frontend ships a ConcatenatedJsonParser (brace-depth/string-aware scanner) and treats the structured agentRes payload as the completion signal, cancelling the reader itself.

   Evidence: apps/frontend/lib/chatStream.ts

5. 05

   Redis does triple duty

   One REDIS_URL backs BullMQ connections, a per-user/per-conversation cache of docId + last 10 responses (TTL 3600, db 2), and the LangGraph RedisSaver checkpointer (db 0), avoiding extra infrastructure.

   Evidence: packages/config/config.ts

6. 06

   LLM-generated conversation titles

   On the first message of a conversation, the chat model is invoked with a detailed title-generation prompt before any assistant answer, and the title is persisted to the conversations table.

   Evidence: apps/backend/src/controllers/chat.controller.ts

The hard parts

• +Stream protocol mismatch: the backend writes concatenated JSON with no delimiter and never calls res.end(), forcing an unusual client-side incremental JSON parser and reader.cancel() as the completion mechanism (apps/frontend/lib/chatStream.ts documents this as an integration blocker, including a missing conversationId event)
• +Multi-tenant vector retrieval: every Qdrant query and delete must filter on metadata.docId + metadata.userId, and correctness depends on chunk metadata set in the worker (apps/queue/src/worker/index.ts)
• +Blocking uploads: createDoc waits up to 120s on the ingestion job, so large PDFs or worker downtime stall requests
• +Citation fidelity: the prompt requires 1-based references arrays with real page numbers extracted from PDFLoader metadata loc.pageNumber and forbids hallucinated ones
• +Async consistency: the Q/A pair is persisted via a fire-and-forget ingest-past-res queue job after streaming, so the chats table can lag the conversation

Known limitations

• −The chat UI is unimplemented: apps/frontend/app/chats/[docId]/page.tsx is an empty file, so the streaming client exists but has no page consuming it (inferred: integration stopped mid-way; ChatSideBar component is built but unused per CONTEXT.md)
• −No tests anywhere in the repo
• −No license file; backend package.json says license ISC but no LICENSE at root
• −README.md is the untouched Turborepo starter boilerplate; real documentation lives in CONTEXT.md
• −No .env.example; env var names (DATABASE_URL, REDIS_URL, GOOGLE_API_KEY, DEEPSEEK_API_KEY, QDRANT_URL, QDRANT_API_KEY, CLERK_*, BUCKETNAME, REGION) only discoverable from code
• −Model name "deepseek-flash" in apps/backend/src/config/ai.ts is not a standard DeepSeek model id, so the chat call likely fails as-is (inferred)
• −No global Express error middleware; dev-server.log sits stray in apps/; no CI config or Docker/compose files present