@astermind/astermind-pro

# Astermind Pro - Premium Features This document lists all premium features extracted from the `astermind-kelm-elm-demo` that are **not included** in the base `@astermind/astermind-elm` package. --- ## Overview The demo showcases **AsterMind Omega**, an advanced RAG (Retrieval-Augmented Generation) system built on top of Astermind ELM. These premium features extend the base ELM capabilities with: - Advanced retrieval and reranking - Sophisticated summarization - Information flow analysis - Production-grade numerical methods - Advanced text processing --- ## 1. Omega RAG System ### 1.1 Omega (`Omega.ts`) **Advanced RAG answer composition with recursive compression** **Features:** - Recursive sentence compression using online ridge regression - Multi-round summarization with weighted sentence selection - Query-aligned sentence scoring with cosine similarity - Lexical bonus for overlapping query terms - Personality modes (neutral, teacher, scientist) - Deterministic, context-locked summarization **Not in base ELM:** Base ELM has no RAG or summarization capabilities. --- ### 1.2 OmegaRR (`OmegaRR.ts`) **Production-grade reranking system with engineered features** **Features:** - **Rich Feature Engineering:** - TF-IDF and BM25 sparse similarity - Heading-query match scores - Jaccard token overlap - Code block detection flags - Structural signals (Go code, links, etc.) - Random projection dense hints - Length heuristics - Prior score integration - **Weak Supervision:** - Automatic label generation from heuristics - Per-query ridge model training - Relevance probability estimation - **MMR (Maximal Marginal Relevance) Filtering:** - Diversity-aware selection - Character budget constraints - Coverage optimization - **Feature Exposure:** - Optional feature vector export - Feature name mapping - Diagnostic utilities **Not in base ELM:** Base ELM has no reranking, feature engineering, or MMR capabilities. --- ### 1.3 OmegaSumDet (`OmegaSumDet.ts`) **Deterministic, intent-aware summarization** **Features:** - **Intent Detection:** - Function, variable, constant detection - Concurrency pattern recognition - Loop detection - **Code-Aware Processing:** - Atomic code block handling - Intent-aware code gating - Code relevance scoring - Query-aligned code inclusion - **Advanced Text Processing:** - Stemmed Dice coefficient for heading alignment - Stopword-aware tokenization - Jaccard deduplication - Section diversity capping - **Deterministic Scoring:** - Normalized feature weights - Explicit tie-breakers - Stable ordering guarantees - Context-locked (no leakage) - **Output Shaping:** - Character budget management - Bullet point formatting - Citation generation - Footer with sources **Not in base ELM:** Base ELM has no summarization, intent detection, or code-aware processing. --- ## 2. Advanced Numerical Methods ### 2.1 KRR - Kernel Ridge Regression (`krr.ts`) **Production-grade ridge regression solver** **Features:** - Cholesky decomposition with adaptive jitter - Conjugate Gradient (CG) fallback for ill-conditioned systems - Symmetry enforcement - Matrix validation (NaN/Inf detection) - Abort signal support for cancellation - Comprehensive diagnostics **Not in base ELM:** Base ELM uses simpler ridge solving without these production features. --- ### 2.2 RFF - Random Fourier Features (`rff.ts`) **RBF kernel approximation via random projections** **Features:** - Deterministic RFF construction - Box-Muller Gaussian sampling - L2 normalization for stability - Efficient cosine/sine feature mapping **Not in base ELM:** Base ELM's KernelELM uses exact kernels or Nyström, but not RFF. --- ### 2.3 OnlineRidge (`online_ridge.ts`) **Online ridge regression with rank-1 updates** **Features:** - Incremental updates via Sherman-Morrison formula - Efficient inverse maintenance - Multi-output support (stacked heads) - No full retraining required **Not in base ELM:** Base OnlineELM uses RLS but not this specific online ridge implementation. --- ### 2.4 Advanced Math Utilities (`math.ts`) **Production-grade numerical operations** **Features:** - Robust vector operations (dot, add, scale, Hadamard) - In-place variants for GC efficiency - Safe exp/log/sigmoid with overflow guards - Stable softmax via log-sum-exp trick - Hyperbolic distance proxy - Vector normalization and clamping - Statistical functions (mean, variance, standardization) - Top-K selection - Formatting utilities **Not in base ELM:** Base ELM has basic math but not these production-grade utilities. --- ## 3. Information Flow Analysis ### 3.1 Transfer Entropy (`infoflow/TransferEntropy.ts`) **Information-theoretic causal analysis** **Features:** - Streaming Transfer Entropy (TE) estimation - Linear-Gaussian approximation - Configurable lag windows - Ridge-regularized regression - Bits/nats reporting - Multi-variable monitoring - InfoFlow graph construction **Use Cases:** - Monitor information flow between RAG components - Query → Score influence tracking - Feature → Relevance analysis - Kept chunks → Summary grounding verification **Not in base ELM:** Base ELM has no information flow or causal analysis capabilities. --- ### 3.2 Transfer Entropy PWS (`infoflow/TransferEntropyPWS.ts`) **Phase-Weighted Stacking variant with importance sampling** **Features:** - **Importance Sampling:** - Rare event detection via tail quantile thresholds - Tail boost weighting for important samples - Time decay for recency weighting - **Path-Weight Sampling (PWS):** - Jittered context histories - Multiple perturbed path averaging - Kernel Density Estimation (KDE) for conditional entropy - Silverman's rule-of-thumb bandwidth selection - **KDE-Based Estimation:** - Product Gaussian kernels - Joint and marginal density estimation - Ridge floor to avoid log(0) issues - Importance-weighted averaging **Not in base ELM:** Advanced variant with importance sampling and PWS not available in base. --- ### 3.3 TE Controller (`infoflow/TEController.ts`) **Closed-loop adaptive control via Transfer Entropy** **Features:** - **Target Band Management:** - Configurable TE target bands per channel - Query→Score, Feature→Score, Kept→Summary monitoring - Optional loop stability guards - **Adaptive Parameter Tuning:** - EMA (Exponential Moving Average) smoothing - Single-knob adjustments per step - Cooldown periods between adjustments - Hard caps on session adjustments - **Knob Control:** - Alpha (sparse/dense mix) - Sigma (kernel bandwidth) - Ridge (regularization) - ProbThresh (reranker threshold) - MMR Lambda (diversity tradeoff) - BudgetChars (answer length) - **Safety Features:** - Minimum sample requirements - Parameter limits enforcement - Step size constraints - Adjustment history tracking **Use Cases:** - Automatic hyperparameter tuning based on information flow - Maintaining optimal TE ranges for system health - Preventing information leakage or overfitting - Adaptive system optimization **Not in base ELM:** No closed-loop control or TE-based adaptation in base. --- ## 4. Advanced Retrieval **NEW:** All retrieval functionality is now available as **standalone, reusable modules** outside of workers! Use `buildIndex()`, `hybridRetrieve()`, and related functions directly in your applications. ### 4.1 Hybrid Retrieval System **Combining sparse and dense methods** **Features:** - **TF-IDF Sparse Retrieval:** - Heading-weighted tokenization - Custom vocabulary management - Stemming support - **Dense Kernel Retrieval:** - Nyström approximation for scalability - Multiple kernel types (RBF, cosine, polynomial) - Landmark-based projection - **Hybrid Scoring:** - Ridge-regularized combination - Alpha/beta mixing parameters - Keyword bonus integration - Tanh clipping for stability **Not in base ELM:** Base ELM has EmbeddingStore but not this hybrid retrieval pipeline. **Available Modules:** - `buildIndex()` - Build vocabulary, IDF, and Nyström landmarks from documents - `hybridRetrieve()` - Perform hybrid retrieval (sparse + dense + keyword bonus) - `toTfidf()` - Compute TF-IDF vectors - `cosineSparse()` - Sparse vector cosine similarity - `projectToDense()` - Project sparse vectors to dense using Nyström landmarks - `parseMarkdownToSections()` - Parse markdown into hierarchical sections - `flattenSections()` - Flatten sections into chunks --- ### 4.2 Nyström Approximation **Efficient kernel computation for large datasets** **Features:** - Landmark selection strategies - Sparse-to-dense projection - Kernel similarity computation - Normalized feature vectors - Configurable landmark count **Not in base ELM:** Base KernelELM has Nyström but not integrated into a full retrieval system. --- ## 5. Advanced Text Processing ### 5.1 Tree-Aware Markdown Parsing **Hierarchical section extraction** **Features:** - Multi-level heading parsing (##, ###, etc.) - Parent-child relationship tracking - Empty parent backfilling - Rich + plain text retention - Code fence preservation - Link handling **Not in base ELM:** Base ELM has basic text encoding but not markdown parsing. --- ### 5.2 Advanced Tokenization **Production-grade text preprocessing** **Features:** - **Stemming:** - Plural folding (ies → y, etc.) - Suffix removal (ization → ize) - Conservative rule-based stemming - Memoization for performance - **Query Expansion:** - Domain-specific term expansion - Context-aware keyword addition - Go-specific expansions - **Code-Aware Splitting:** - Fenced code block detection - Atomic code block handling - Mixed text/code processing **Not in base ELM:** Base ELM has UniversalEncoder but not this advanced preprocessing. --- ### 5.3 ELM Scorer (`elm_scorer.ts`) **Custom ELM-based relevance scoring** **Features:** - Random hidden layer initialization - GELU activation approximation - Online ridge output layer - Partial fit support - Deterministic seeding **Not in base ELM:** This is a custom implementation, though base ELM could be used similarly. --- ## 6. Auto-Tuning System **NEW:** Auto-tuning is now available as a **standalone function** outside of workers! Use `autoTune()` directly in your applications. ### 6.1 Hyperparameter Optimization **Automated configuration search** **Features:** - Random search with refinement - Jaccard-based evaluation metric - Ridge-aware parameter exploration - Caching for efficiency - Penalty functions for complexity - Real-time progress reporting **Parameters Tuned:** - Alpha (sparse/dense mix) - Beta (keyword bonus) - Sigma (kernel bandwidth) - Kernel type - Vocabulary size - Landmark count - Prefilter size - Top-K selection - Heading weights - Chunk/overlap sizes - Ridge regularization **Not in base ELM:** Base ELM has no auto-tuning capabilities. **Available Functions:** - `autoTune()` - Automated hyperparameter optimization - `sampleQueriesFromCorpus()` - Generate synthetic queries for tuning - `penalty()` - Compute complexity penalty for configurations - `jaccard()` - Calculate Jaccard similarity between index sets --- ## 7. Model Persistence **NEW:** Model serialization is now available as **standalone functions** outside of workers! Use `exportModel()` and `importModel()` directly in your applications. ### 7.1 Serialized Model Format **Complete model export/import** **Features:** - Full state serialization - Version tracking - Optional dense vector storage - Checksum generation - Settings snapshot - Vocabulary + IDF preservation - Nyström landmark storage - Chunk metadata retention **Not in base ELM:** Base ELM has JSON import/export but not this comprehensive format. **Available Functions:** - `exportModel()` - Export complete model state to serialized format - `importModel()` - Import model from serialized format - `quickHash()` - Generate deterministic hash for model verification --- ## 8. Worker Architecture ### 8.1 Web Worker Pipeline **Background processing for RAG** **Features:** - Non-blocking retrieval - Async model loading - Progress reporting - Error handling - Message-based API - State management **Not in base ELM:** Base ELM has ELMWorker but not this full RAG pipeline. --- ## 9. Advanced ELM Variants ### 9.1 Deep ELM Pro (`elm/deep-elm-pro.ts`) **Improved multi-layer ELM with advanced training strategies** **Improvements over Base DeepELM:** | Feature | Base DeepELM | DeepELMPro | |---------|--------------|------------| | **Pretraining** | ❌ None | ✅ Autoencoder pretraining for each layer | | **Training Strategy** | Joint only | ✅ Layer-wise or Joint (configurable) | | **Regularization** | ❌ None | ✅ L1/L2/Elastic Net regularization | | **Batch Normalization** | ❌ None | ✅ Optional batch normalization between layers | | **Dropout** | ❌ None | ✅ Optional dropout with configurable rate | | **Training Flexibility** | Basic | ✅ Advanced with multiple strategies | **Key Features:** - **Autoencoder Pretraining**: Each layer can be pretrained as an autoencoder to learn better feature representations before supervised training - **Layer-wise Training**: Train layers sequentially for more stable learning and better feature extraction - **Regularization**: L1, L2, and Elastic Net regularization to prevent overfitting and improve generalization - **Batch Normalization**: Normalize activations between layers for faster convergence and training stability - **Dropout**: Randomly drop neurons during training to reduce overfitting - **Flexible Training Modes**: Choose between layer-wise sequential training or joint training based on your data **Not in base ELM:** Base DeepELM has basic multi-layer support but lacks these advanced training strategies, regularization, and normalization techniques. **Use Cases:** - Complex pattern recognition requiring deep feature hierarchies - High-dimensional data classification - When base DeepELM overfits or doesn't converge well - Production systems requiring robust, generalizable models --- ### 9.2 Multi-Kernel ELM (`elm/multi-kernel-elm.ts`) **Combines multiple kernel types for improved accuracy** **Features:** - Weighted combination of multiple kernels (RBF, linear) - Automatic kernel weight learning based on validation performance - Multiple kernel types support - Ridge-regularized combination **Not in base ELM:** Base KernelELM uses a single kernel type. Multi-Kernel ELM combines multiple kernels for better performance. --- ### 9.3 Online Kernel ELM (`elm/online-kernel-elm.ts`) **Real-time learning for streaming data** **Features:** - Incremental kernel matrix updates - Sliding window with forgetting mechanisms - Adaptive landmark selection - Real-time prediction **Not in base ELM:** Base KernelELM requires batch training. Online Kernel ELM supports streaming updates. --- ### 9.4 Multi-Task ELM (`elm/multi-task-elm.ts`) **Joint learning across related tasks** **Features:** - Shared feature extraction layer - Task-specific output layers - Task weighting for importance - Joint optimization **Not in base ELM:** Base ELM trains one task at a time. Multi-Task ELM learns multiple related tasks simultaneously. --- ### 9.5 Sparse ELM (`elm/sparse-elm.ts`) **Efficiency and interpretability for high-dimensional data** **Features:** - L1/L2/Elastic net regularization - Weight pruning for sparsity - Feature importance ranking - Interpretable models **Not in base ELM:** Base ELM has no built-in sparsity mechanisms. Sparse ELM provides regularization and feature selection. --- ## Summary: Premium Feature Categories ### Core Premium Features: 1. ✅ **Omega RAG System** - Complete RAG pipeline 2. ✅ **OmegaRR Reranking** - Production reranking with MMR 3. ✅ **OmegaSumDet** - Intent-aware summarization 4. ✅ **Transfer Entropy** - Information flow analysis 5. ✅ **Hybrid Retrieval** - Sparse + dense combination (**NEW: Standalone modules!**) 6. ✅ **Auto-Tuning** - Hyperparameter optimization (**NEW: Standalone function!**) 7. ✅ **Advanced Math** - Production-grade numerics 8. ✅ **Tree-Aware Parsing** - Hierarchical markdown processing (**NEW: Standalone functions!**) 9. ✅ **Model Serialization** - Export/import models (**NEW: Standalone functions!**) 10. ✅ **Tokenization & Utilities** - Reusable text processing (**NEW: Standalone functions!**) 11. ✅ **Advanced ELM Variants** - 5 premium variants (**NEW!**) ### Advanced Algorithms: - KRR (Kernel Ridge Regression with CG fallback) - RFF (Random Fourier Features) - OnlineRidge (Rank-1 updates) - Nyström approximation - MMR filtering - Weak supervision ### Production Features: - Robust error handling - Matrix validation - Abort signals - Comprehensive diagnostics - Model versioning - Checksums --- ## Integration Notes These premium features are designed to work **on top of** `@astermind/astermind-elm`: - Uses base ELM's `Tokenizer`, `TFIDFVectorizer`, `ELM`, `OnlineELM` - Extends with premium retrieval, reranking, and summarization - Adds information flow monitoring - Provides production-grade numerical methods --- ## License & Usage These premium features are part of **Astermind Pro** and are not included in the MIT-licensed Community Edition (`@astermind/astermind-elm`). --- *Last updated: 2025-01-16*