Semantic Search

Collect all source documents (PDFs, web pages, notes) and split them into semantically meaningful chunks (e.g., paragraphs or sections). Each chunk should be self-contained and sized to fit within the embedding model's token limit. This step ensures that later search retrieves precise, relevant passages rather than entire documents.

Why NucliaDB: NucliaDB provides automated document ingestion and indexing, which directly covers both document parsing and chunking for multi-modal documents.

Pass each chunk through a text embedding model (e.g., OpenAI text-embedding-3-small, sentence-transformers) to produce dense vector representations. Store the vectors in a vector database (e.g., Pinecone, Weaviate, FAISS) along with the chunk text and metadata. This creates a searchable index that captures semantic meaning.

Why Superlinked: Superlinked explicitly generates text embeddings for semantic search and supports similarity search, covering both embedding generation and storage needs.

Take the raw user query and preprocess it: expand abbreviations, correct typos, and optionally generate multiple query variations (e.g., synonyms, rephrasings) to improve recall. For complex queries, extract key entities or intents. This step ensures the search engine understands the user's true information need.

Why Voyage AI: Voyage AI offers reranking for improved relevance and can enhance query enrichment through embedding-based analysis.

Embed the enriched query using the same embedding model, then perform a nearest-neighbor search in the vector database to retrieve the top-K most similar chunks (e.g., K=20). Optionally apply metadata filters (e.g., date range, category) to narrow results. This step returns a candidate pool of relevant passages.

Why Weaviate: Weaviate is a dedicated vector search and semantic search engine, directly providing the vector database query API needed.

Apply a cross-encoder or lightweight classifier to re-rank the candidate chunks by relevance to the original query. Optionally classify each result into predefined categories (e.g., 'answer', 'supporting evidence', 'irrelevant'). This step boosts precision and organizes results for the end user.

Why Cohere: Cohere provides semantic search and document summarization, and its rerank API is a standard cross-encoder for re-ranking results.

Format the final results for the user: display each chunk's text, its source document title, and a direct link or page number. If the search is part of a Q&A system, optionally generate a synthesized answer using an LLM that cites the retrieved chunks. This step delivers actionable, trustworthy information.

Why Onyx AI (formerly Danswer AI): Onyx AI (formerly Danswer AI) provides enterprise knowledge search and AI-powered Q&A over company data, ideal for presenting results with context and citations.