Problem

Saved links, videos, code snippets and notes are hard to recall later because finding them requires remembering exact wording; the project wants semantic retrieval over personal knowledge (README.md, lines 21-24).

What it does

Users create collections and add content items typed as WEBPAGE, YOUTUBE, CODE or NOTE (lib/prisma/schema.prisma, lines 50-54). On save, the app fetches a YouTube transcript or scrapes a URL with axios+cheerio (lib/Utils/utils.ts, lines 11-44), asks Gemini 2.0 Flash for a ~100-word JSON summary (app/api/[collectionID]/content/route.ts, lines 31-52), embeds the summary with Gemini embedding-001, and stores it in a pgvector column (route.ts lines 60-81; schema.prisma lines 11, 44). Search embeds the query and runs a raw cosine-distance query ordering by 1 - (embedding <-> vector) with LIMIT 5 (app/api/[collectionID]/search/route.ts, lines 27-44). A Clerk webhook (Svix-verified) syncs user create/delete into Prisma (app/api/webhook/register/route.ts).

Architecture

[web] Next.js 15 dashboard (React 19, Tailwind 4, motion, lucide) (client)
[api] Next.js API routes (collections, content, search, webhook) (api)
[auth] Clerk auth + Svix-signed webhook (external)
[gemini] Gemini 2.0 Flash (summarization) + embedding-001 (model)
[fetchers] YouTube transcript + axios/cheerio URL scraper (worker)
[db] PostgreSQL + pgvector via Prisma (store)

web ──▶ auth · useAuth/useUser; RedirectToSignUp (app/dashboard/page.tsx)
auth ──▶ api · user.created/deleted webhook -> Prisma user sync
web ──▶ api · axios calls to /api/collections/:userId and /api/:collectionID/content (hardcoded localhost:3000, components/MainDashboard.tsx line 25)
api ──▶ fetchers · extractAdditionalContent for YOUTUBE/WEBPAGE types
api ──▶ gemini · summary prompt + embedContent(summary)
api ──▶ db · raw SQL INSERT with vector; cosine-distance search query

Simplified — box-and-arrow sketch of the repo structure, not a system diagram.

Decisions

1. 01

   Embed the LLM summary, not the raw content

   Summary is a compact ~100-word distillation of the transcript/scraped page, keeping the vector small and search focused on meaning rather than full-page noise; embedding is generated from `summary` (app/api/[collectionID]/content/route.ts line 60).

   Evidence: app/api/[collectionID]/content/route.ts

2. 02

   pgvector cosine search via Prisma $queryRaw

   Prisma has no native vector API, so raw SQL with parameterized `1 - ("embedding" <-> $vector)` ordering is used (app/api/[collectionID]/search/route.ts lines 30-44); the vector extension is enabled via postgresqlExtensions preview feature (lib/prisma/schema.prisma lines 3, 11).

   Evidence: app/api/[collectionID]/search/route.ts

3. 03

   Type-specific ingestion pipeline

   YouTube links are resolved to transcripts with youtube-transcript and webpages are stripped of script/nav/footer chrome with cheerio before summarizing, so summaries rely on the actual content (lib/Utils/utils.ts lines 29-62).

   Evidence: lib/Utils/utils.ts

4. 04

   Strict JSON-only summary prompt

   The prompt demands a bare {"summary":...} object with escaping rules and fallback warnings when a YouTube link is mis-typed, then strips ```json fences and JSON.parse's the response (route.ts lines 31-66).

   Evidence: app/api/[collectionID]/content/route.ts

5. 05

   Clerk as identity backbone

   clerkMiddleware guards routes (middleware.ts), ClerkProvider wraps the app (app/layout.tsx line 27), and a Svix-verified webhook maintains the local User table keyed by clerkId (app/api/webhook/register/route.ts lines 29-62).

   Evidence: app/api/webhook/register/route.ts

6. 06

   Per-tenant filtering on clerkId

   Every query (collections, contents, search) is scoped by the clerkId passed as a query param, keeping data partitioned per user, though this is client-supplied rather than derived server-side.

   Evidence: app/api/collections/[userId]/route.ts lines 20-23

The hard parts

• +Vector insert and cosine search had to be done with raw SQL ($executeRaw/$queryRaw) because Prisma's vector support is an Unsupported column (lib/prisma/schema.prisma line 44)
• +Coaxing Gemini into clean parseable JSON required an elaborate prompt plus post-hoc fence stripping (route.ts lines 31-66)
• +Reliable text extraction from third-party sources: YouTube transcript fetching and cheerio-based page scraping both swallow errors into returned error strings (lib/Utils/utils.ts lines 18-20, 41-43)
• +Clerk user provisioning must run before any collection use, hence the verified webhook route

Known limitations

• −No automated tests, no CI, no Dockerfile/compose, no license file
• −Client fetches are hardcoded to http://localhost:3000 (components/MainDashboard.tsx lines 25, 39), so it only works in local dev
• −API routes trust a userID query param instead of verifying the Clerk session server-side, so data isolation depends on the caller (collections GET filters only by query-param clerkId; app/api/collections/[userId]/route.ts lines 8-31)
• −Search results are scoped to a single collection, not across all collections as the README implies (README.md line 24 vs search route WHERE clause)
• −Gemini API key falls back to the placeholder "12345" (lib/ai/geminiUtils.ts line 3), and errors are returned raw to clients including debug params (search/route.ts line 20)
• −Embedding column is untyped (Unsupported("vector")) with no dimension or index declared; no IVFFlat/HNSW index, so search degrades at scale (inferred from schema.prisma line 44)
• −PUT only updates title/description; editing main content never regenerates the summary or embedding (route.ts lines 167-179)
• −package.json is still named "authentication-demo" and both pnpm-lock.yaml and package-lock.json are committed