# Neural Information Retrieval (IR) How search engines and AI systems find and retrieve information. This collection explains the mechanics behind semantic search, how AI assistants cite sources, and why certain content surfaces in results. Covers both the theory (how these systems work) and practice (how to build them). ## Highlights Most referenced resources in this collection: - [[Retrieval Augmented Generation (RAG)]] (26 unique files) - Core technique for LLM-based question answering - [[Hypothetical Document Embeddings (HyDE)]] (18 unique files) - Advanced retrieval technique - [[Information Retrieval (IR) System]] (16 unique files) - System architecture and components - [[Information Retrieval (IR)]] (15 unique files) - Foundational concepts - [[Vector Database]] (14 unique files) - Storage for embeddings - [[Re-ranker]] (14 unique files) - Result re-ranking for improved relevance - [[Semantic Search]] (14 unique files) - Neural/embedding-based search - [[Vector Embeddings]] (14 unique files) - Core embedding concepts - [[Composite Embedding]] (12 unique files) - Multi-modal embedding techniques - [[Bi-encoder]] (11 unique files) - Independent query/document encoding - [[Query Transformation]] (11 unique files) - Improving query effectiveness - [[Large Language Model (LLM)]] (5 unique files) - Foundation models for NLP - [[Cosine Similarity]] (5 unique files) - Vector similarity metric - [[Natural Language Processing (NLP)]] (5 unique files) - Language understanding fundamentals ## Core Concepts - [[Information Retrieval (IR)]] - Foundational principles - [[Information Retrieval (IR) System]] - Architecture and design - [[Information Retrieval (IR) Tech Stack]] - Technology choices - [[What AI Engineers Should Know about Search]] - Practical overview - [[IR Unit]] - Basic building blocks - [[Large Language Model (LLM)]] - Foundation models for natural language processing - [[Natural Language Processing (NLP)]] - Language understanding and processing ## Search Types & Techniques ### Search Paradigms - [[Semantic Search]] - Neural/embedding-based search - [[Lexical Search]] - Traditional keyword search - [[Hybrid Search]] - Combining semantic and lexical approaches - [[Symmetric & Asymmetric Semantic Search]] (6 unique files) - Search modality patterns - [[Asymmetric Semantic Search]] - Query-document asymmetric search - [[Search Engine Results Page (SERP)]] - Search result presentation ### Query Optimization - [[Query Transformation]] - Improving query effectiveness - [[Query Embedding]] (8 unique files) - Query vector representation - [[Query Expansion]] (7 unique files) - Broadening query scope - [[Query Augmentation]] (3 unique files) - Enhancing queries - [[Query Classification]] - Understanding query types - [[Query Routing]] - Directing queries to appropriate systems - [[Query Relaxation]] - Loosening query constraints for better recall - [[Types of Queries]] - Query taxonomy - [[Query2doc]] - Query expansion technique - [[Relevance Feedback]] - Iterative query refinement - [[TextRank]] - Graph-based keyword extraction - [[KeyBERT]] - BERT-based keyword extraction - [[GDELT Query Generation Rules]] - GDELT-specific query generation ### Advanced Retrieval - [[Hypothetical Document Embeddings (HyDE)]] - Document generation for better matching - [[Re-ranker]] - Post-retrieval ranking - [[Reranking Methods]] - Techniques for result reranking - [[Combining Bi- and Cross-Encoders]] - Hybrid encoding strategies - [[Retrieval Method]] (3 unique files) - Retrieval methodologies ## Embeddings & Vectors ### Vector Representations - [[Vector Embeddings]] - Core embedding concepts - [[Composite Embedding]] (12 unique files) - Multi-modal embedding techniques - [[Document Embeddings]] (7 unique files) - Document vector representation - [[Query Embedding]] (8 unique files) - Query vector representation - [[Embedding Model]] - Models that generate embeddings - [[Embedding Projections]] (3 unique files) - Visualization and dimensionality reduction - [[EMBEDDING PROJECTION]] - Embedding projection techniques - [[Dense Vectors ("Embeddings")]] - Dense vector representations - [[Sparse Vectors ("Embeddings")]] - Sparse vector representations - [[Two Tower Embeddings ("Learned Embeddings")]] - Dual encoder architectures - [[LLMs and Embeddings]] (5 unique files) - LLM-based embeddings - [[Common Uses of Vector Embeddings]] - Application patterns - [[Cosine Similarity]] - Vector similarity metric for embeddings ### Encoders - [[Bi-encoder]] - Independent query/document encoding - [[Cross-encoder]] - Joint query/document encoding - [[SPLADE]] - Sparse learned representations ### Infrastructure - [[Vector Database]] - Storage for embeddings - [[Data Structures - Scalar vs Vector vs Matrix vs Tensor]] - Mathematical foundations ## RAG & Generation - [[Retrieval Augmented Generation (RAG)]] - Core RAG concepts - [[Building Production-Grade LLM or RAG Apps]] - Production considerations - [[Pinecone RAG Study]] - RAG implementation case study - [[RECOMP]] - Compression for RAG - [[Multi-Text Generation Integration (MUGI)]] - Multi-text generation - [[Rewrite Texts by LLM for Self-sufficiency Pre-chunking]] - Content preprocessing - [[Bias and Knowledge Conflicts in Retrieval-Augmented Language Models (RALM)]] - RALM bias research - [[sparse mixture-of-experts (MoE) network]] - Sparse MoE architectures ## Evaluation & Metrics - [[IR Evaluation Metrics]] - Overview of metrics - [[IR Precision]] - Precision measurement - [[IR Recall]] - Recall measurement - [[IR F1 Score]] - F1 score calculation - [[IR Accuracy]] - Accuracy measurement ## Traditional IR Methods - [[Okapi BM25]] - Classic ranking function - [[BM25S]] - BM25 variant implementation - [[Term Frequency-Inverse Document Frequency (TF-IDF)]] - Traditional weighting - [[TF-IDF]] - Implementation details - [[Bag-of-documents Model]] - Document representation ## Clustering & Topic Modeling ### Clustering Algorithms - [[K-means]] - Partitional clustering - [[HDBSCAN]] - Density-based clustering - [[Partitional Clustering]] - Clustering category - [[Identifying the cluster centroid that is closest to the new data point.]] - Cluster assignment ### Cluster Evaluation - [[The Elbow Method]] - Optimal k determination - [[The Gap Statistic]] - Statistical k determination - [[The Silhouette Method]] - Cluster quality assessment - [[Combined Methods to Determine n_observations for k_means Clustering]] - Multi-method approach ### Topic Modeling - [[Topic Modeling]] - Overview of topic modeling approaches - [[BERTopic]] - Neural topic modeling - [[BERTopic Contribution]] - Contributing to BERTopic - [[Latent Dirichlet Allocation (LDA)]] - Classical topic modeling ## Data Processing - [[Text Normalization]] (8 unique files) - Text preprocessing and cleaning - [[Text Chunking]] (6 unique files) - Document segmentation strategies - [[Data Ingestion & Processing]] - Data pipeline fundamentals - [[Extract Transform Load (ETL)]] - ETL processes - [[Metadata Enrichment]] - Adding context to data - [[Small-to-big]] - Chunking strategy - [[Social Network Analysis]] - Network analysis and narrative extraction from text corpora ## Infrastructure & Tools - [[database operations]] - Database fundamentals - [[Database Schema]] - Schema design - [[Langchain]] - LLM application framework - [[OP Stack]] - Infrastructure stack - [[Querysets]] (3 unique files) - Database query abstractions - [[Elasticsearch]] - Search and analytics engine - [[Querying Pinecone by Cluster Labels]] - Pinecone cluster queries - [[Hardware Requirements for Running Open-Source LLMs Locally]] - LLM hardware specs - [[Information Retrieval System.canvas]] - Visual system architecture