[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M70GGEWBGR4V4XQVM5X3H0YK":3,"skill-same:01M70GGEWBGR4V4XQVM5X3H0YK":67},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":48,"ownerHtmlUrl":49,"ownerBlogUrl":50,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M70GGEWBGR4V4XQVM5X3H0YK","vector-index-tuning",null,"Optimize vector index performance for latency, recall, and memory. Use when tuning HNSW parameters, selecting quantization strategies, or scaling vector search infrastructure.","shengdabai\u002FTony-Claude-Code-Skills","shengdabai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F116417419?v=4","01M70F6D2WY5PQV5A92BKYJQEW","Tony Sheng","01M70F6D3GAAAYZ5H4CC1G8VXM",3,"MIT","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fvector-index-tuning\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"vector index tuning","tuning","index","vector","infrastructure","HNSW","Vector Search","recall","Quantization","optimize","strategies","when","use","search","performance","for","memory","latency","scaling","2026-11-10 10:12:30",false,"---\nname: vector-index-tuning\ndescription: Optimize vector index performance for latency, recall, and memory. Use when tuning HNSW parameters, selecting quantization strategies, or scaling vector search infrastructure.\n---\n\n# Vector Index Tuning\n\nGuide to optimizing vector indexes for production performance.\n\n## When to Use This Skill\n\n- Tuning HNSW parameters\n- Implementing quantization\n- Optimizing memory usage\n- Reducing search latency\n- Balancing recall vs speed\n- Scaling to billions of vectors\n\n## Core Concepts\n\n### 1. Index Type Selection\n\n```\nData Size           Recommended Index\n────────────────────────────────────────\n\u003C 10K vectors  →    Flat (exact search)\n10K - 1M       →    HNSW\n1M - 100M      →    HNSW + Quantization\n> 100M         →    IVF + PQ or DiskANN\n```\n\n### 2. HNSW Parameters\n\n| Parameter | Default | Effect |\n|-----------|---------|--------|\n| **M** | 16 | Connections per node, ↑ = better recall, more memory |\n| **efConstruction** | 100 | Build quality, ↑ = better index, slower build |\n| **efSearch** | 50 | Search quality, ↑ = better recall, slower search |\n\n### 3. Quantization Types\n\n```\nFull Precision (FP32): 4 bytes × dimensions\nHalf Precision (FP16): 2 bytes × dimensions\nINT8 Scalar:           1 byte × dimensions\nProduct Quantization:  ~32-64 bytes total\nBinary:                dimensions\u002F8 bytes\n```\n\n## Templates\n\n### Template 1: HNSW Parameter Tuning\n\n```python\nimport numpy as np\nfrom typing import List, Tuple\nimport time\n\ndef benchmark_hnsw_parameters(\n    vectors: np.ndarray,\n    queries: np.ndarray,\n    ground_truth: np.ndarray,\n    m_values: List[int] = [8, 16, 32, 64],\n    ef_construction_values: List[int] = [64, 128, 256],\n    ef_search_values: List[int] = [32, 64, 128, 256]\n) -> List[dict]:\n    \"\"\"Benchmark different HNSW configurations.\"\"\"\n    import hnswlib\n\n    results = []\n    dim = vectors.shape[1]\n    n = vectors.shape[0]\n\n    for m in m_values:\n        for ef_construction in ef_construction_values:\n            # Build index\n            index = hnswlib.Index(space='cosine', dim=dim)\n            index.init_index(max_elements=n, M=m, ef_construction=ef_construction)\n\n            build_start = time.time()\n            index.add_items(vectors)\n            build_time = time.time() - build_start\n\n            # Get memory usage\n            memory_bytes = index.element_count * (\n                dim * 4 +  # Vector storage\n                m * 2 * 4  # Graph edges (approximate)\n            )\n\n            for ef_search in ef_search_values:\n                index.set_ef(ef_search)\n\n                # Measure search\n                search_start = time.time()\n                labels, distances = index.knn_query(queries, k=10)\n                search_time = time.time() - search_start\n\n                # Calculate recall\n                recall = calculate_recall(labels, ground_truth, k=10)\n\n                results.append({\n                    \"M\": m,\n                    \"ef_construction\": ef_construction,\n                    \"ef_search\": ef_search,\n                    \"build_time_s\": build_time,\n                    \"search_time_ms\": search_time * 1000 \u002F len(queries),\n                    \"recall@10\": recall,\n                    \"memory_mb\": memory_bytes \u002F 1024 \u002F 1024\n                })\n\n    return results\n\n\ndef calculate_recall(predictions: np.ndarray, ground_truth: np.ndarray, k: int) -> float:\n    \"\"\"Calculate recall@k.\"\"\"\n    correct = 0\n    for pred, truth in zip(predictions, ground_truth):\n        correct += len(set(pred[:k]) & set(truth[:k]))\n    return correct \u002F (len(predictions) * k)\n\n\ndef recommend_hnsw_params(\n    num_vectors: int,\n    target_recall: float = 0.95,\n    max_latency_ms: float = 10,\n    available_memory_gb: float = 8\n) -> dict:\n    \"\"\"Recommend HNSW parameters based on requirements.\"\"\"\n\n    # Base recommendations\n    if num_vectors \u003C 100_000:\n        m = 16\n        ef_construction = 100\n    elif num_vectors \u003C 1_000_000:\n        m = 32\n        ef_construction = 200\n    else:\n        m = 48\n        ef_construction = 256\n\n    # Adjust ef_search based on recall target\n    if target_recall >= 0.99:\n        ef_search = 256\n    elif target_recall >= 0.95:\n        ef_search = 128\n    else:\n        ef_search = 64\n\n    return {\n        \"M\": m,\n        \"ef_construction\": ef_construction,\n        \"ef_search\": ef_search,\n        \"notes\": f\"Estimated for {num_vectors:,} vectors, {target_recall:.0%} recall\"\n    }\n```\n\n### Template 2: Quantization Strategies\n\n```python\nimport numpy as np\nfrom typing import Optional\n\nclass VectorQuantizer:\n    \"\"\"Quantization strategies for vector compression.\"\"\"\n\n    @staticmethod\n    def scalar_quantize_int8(\n        vectors: np.ndarray,\n        min_val: Optional[float] = None,\n        max_val: Optional[float] = None\n    ) -> Tuple[np.ndarray, dict]:\n        \"\"\"Scalar quantization to INT8.\"\"\"\n        if min_val is None:\n            min_val = vectors.min()\n        if max_val is None:\n            max_val = vectors.max()\n\n        # Scale to 0-255 range\n        scale = 255.0 \u002F (max_val - min_val)\n        quantized = np.clip(\n            np.round((vectors - min_val) * scale),\n            0, 255\n        ).astype(np.uint8)\n\n        params = {\"min_val\": min_val, \"max_val\": max_val, \"scale\": scale}\n        return quantized, params\n\n    @staticmethod\n    def dequantize_int8(\n        quantized: np.ndarray,\n        params: dict\n    ) -> np.ndarray:\n        \"\"\"Dequantize INT8 vectors.\"\"\"\n        return quantized.astype(np.float32) \u002F params[\"scale\"] + params[\"min_val\"]\n\n    @staticmethod\n    def product_quantize(\n        vectors: np.ndarray,\n        n_subvectors: int = 8,\n        n_centroids: int = 256\n    ) -> Tuple[np.ndarray, dict]:\n        \"\"\"Product quantization for aggressive compression.\"\"\"\n        from sklearn.cluster import KMeans\n\n        n, dim = vectors.shape\n        assert dim % n_subvectors == 0\n        subvector_dim = dim \u002F\u002F n_subvectors\n\n        codebooks = []\n        codes = np.zeros((n, n_subvectors), dtype=np.uint8)\n\n        for i in range(n_subvectors):\n            start = i * subvector_dim\n            end = (i + 1) * subvector_dim\n            subvectors = vectors[:, start:end]\n\n            kmeans = KMeans(n_clusters=n_centroids, random_state=42)\n            codes[:, i] = kmeans.fit_predict(subvectors)\n            codebooks.append(kmeans.cluster_centers_)\n\n        params = {\n            \"codebooks\": codebooks,\n            \"n_subvectors\": n_subvectors,\n            \"subvector_dim\": subvector_dim\n        }\n        return codes, params\n\n    @staticmethod\n    def binary_quantize(vectors: np.ndarray) -> np.ndarray:\n        \"\"\"Binary quantization (sign of each dimension).\"\"\"\n        # Convert to binary: positive = 1, negative = 0\n        binary = (vectors > 0).astype(np.uint8)\n\n        # Pack bits into bytes\n        n, dim = vectors.shape\n        packed_dim = (dim + 7) \u002F\u002F 8\n\n        packed = np.zeros((n, packed_dim), dtype=np.uint8)\n        for i in range(dim):\n            byte_idx = i \u002F\u002F 8\n            bit_idx = i % 8\n            packed[:, byte_idx] |= (binary[:, i] \u003C\u003C bit_idx)\n\n        return packed\n\n\ndef estimate_memory_usage(\n    num_vectors: int,\n    dimensions: int,\n    quantization: str = \"fp32\",\n    index_type: str = \"hnsw\",\n    hnsw_m: int = 16\n) -> dict:\n    \"\"\"Estimate memory usage for different configurations.\"\"\"\n\n    # Vector storage\n    bytes_per_dimension = {\n        \"fp32\": 4,\n        \"fp16\": 2,\n        \"int8\": 1,\n        \"pq\": 0.05,  # Approximate\n        \"binary\": 0.125\n    }\n\n    vector_bytes = num_vectors * dimensions * bytes_per_dimension[quantization]\n\n    # Index overhead\n    if index_type == \"hnsw\":\n        # Each node has ~M*2 edges, each edge is 4 bytes (int32)\n        index_bytes = num_vectors * hnsw_m * 2 * 4\n    elif index_type == \"ivf\":\n        # Inverted lists + centroids\n        index_bytes = num_vectors * 8 + 65536 * dimensions * 4\n    else:\n        index_bytes = 0\n\n    total_bytes = vector_bytes + index_bytes\n\n    return {\n        \"vector_storage_mb\": vector_bytes \u002F 1024 \u002F 1024,\n        \"index_overhead_mb\": index_bytes \u002F 1024 \u002F 1024,\n        \"total_mb\": total_bytes \u002F 1024 \u002F 1024,\n        \"total_gb\": total_bytes \u002F 1024 \u002F 1024 \u002F 1024\n    }\n```\n\n### Template 3: Qdrant Index Configuration\n\n```python\nfrom qdrant_client import QdrantClient\nfrom qdrant_client.http import models\n\ndef create_optimized_collection(\n    client: QdrantClient,\n    collection_name: str,\n    vector_size: int,\n    num_vectors: int,\n    optimize_for: str = \"balanced\"  # \"recall\", \"speed\", \"memory\"\n) -> None:\n    \"\"\"Create collection with optimized settings.\"\"\"\n\n    # HNSW configuration based on optimization target\n    hnsw_configs = {\n        \"recall\": models.HnswConfigDiff(m=32, ef_construct=256),\n        \"speed\": models.HnswConfigDiff(m=16, ef_construct=64),\n        \"balanced\": models.HnswConfigDiff(m=16, ef_construct=128),\n        \"memory\": models.HnswConfigDiff(m=8, ef_construct=64)\n    }\n\n    # Quantization configuration\n    quantization_configs = {\n        \"recall\": None,  # No quantization for max recall\n        \"speed\": models.ScalarQuantization(\n            scalar=models.ScalarQuantizationConfig(\n                type=models.ScalarType.INT8,\n                quantile=0.99,\n                always_ram=True\n            )\n        ),\n        \"balanced\": models.ScalarQuantization(\n            scalar=models.ScalarQuantizationConfig(\n                type=models.ScalarType.INT8,\n                quantile=0.99,\n                always_ram=False\n            )\n        ),\n        \"memory\": models.ProductQuantization(\n            product=models.ProductQuantizationConfig(\n                compression=models.CompressionRatio.X16,\n                always_ram=False\n            )\n        )\n    }\n\n    # Optimizer configuration\n    optimizer_configs = {\n        \"recall\": models.OptimizersConfigDiff(\n            indexing_threshold=10000,\n            memmap_threshold=50000\n        ),\n        \"speed\": models.OptimizersConfigDiff(\n            indexing_threshold=5000,\n            memmap_threshold=20000\n        ),\n        \"balanced\": models.OptimizersConfigDiff(\n            indexing_threshold=20000,\n            memmap_threshold=50000\n        ),\n        \"memory\": models.OptimizersConfigDiff(\n            indexing_threshold=50000,\n            memmap_threshold=10000  # Use disk sooner\n        )\n    }\n\n    client.create_collection(\n        collection_name=collection_name,\n        vectors_config=models.VectorParams(\n            size=vector_size,\n            distance=models.Distance.COSINE\n        ),\n        hnsw_config=hnsw_configs[optimize_for],\n        quantization_config=quantization_configs[optimize_for],\n        optimizers_config=optimizer_configs[optimize_for]\n    )\n\n\ndef tune_search_parameters(\n    client: QdrantClient,\n    collection_name: str,\n    target_recall: float = 0.95\n) -> dict:\n    \"\"\"Tune search parameters for target recall.\"\"\"\n\n    # Search parameter recommendations\n    if target_recall >= 0.99:\n        search_params = models.SearchParams(\n            hnsw_ef=256,\n            exact=False,\n            quantization=models.QuantizationSearchParams(\n                ignore=True,  # Don't use quantization for search\n                rescore=True\n            )\n        )\n    elif target_recall >= 0.95:\n        search_params = models.SearchParams(\n            hnsw_ef=128,\n            exact=False,\n            quantization=models.QuantizationSearchParams(\n                ignore=False,\n                rescore=True,\n                oversampling=2.0\n            )\n        )\n    else:\n        search_params = models.SearchParams(\n            hnsw_ef=64,\n            exact=False,\n            quantization=models.QuantizationSearchParams(\n                ignore=False,\n                rescore=False\n            )\n        )\n\n    return search_params\n```\n\n### Template 4: Performance Monitoring\n\n```python\nimport time\nfrom dataclasses import dataclass\nfrom typing import List\nimport numpy as np\n\n@dataclass\nclass SearchMetrics:\n    latency_p50_ms: float\n    latency_p95_ms: float\n    latency_p99_ms: float\n    recall: float\n    qps: float\n\n\nclass VectorSearchMonitor:\n    \"\"\"Monitor vector search performance.\"\"\"\n\n    def __init__(self, ground_truth_fn=None):\n        self.latencies = []\n        self.recalls = []\n        self.ground_truth_fn = ground_truth_fn\n\n    def measure_search(\n        self,\n        search_fn,\n        query_vectors: np.ndarray,\n        k: int = 10,\n        num_iterations: int = 100\n    ) -> SearchMetrics:\n        \"\"\"Benchmark search performance.\"\"\"\n        latencies = []\n\n        for _ in range(num_iterations):\n            for query in query_vectors:\n                start = time.perf_counter()\n                results = search_fn(query, k=k)\n                latency = (time.perf_counter() - start) * 1000\n                latencies.append(latency)\n\n        latencies = np.array(latencies)\n        total_queries = num_iterations * len(query_vectors)\n        total_time = sum(latencies) \u002F 1000  # seconds\n\n        return SearchMetrics(\n            latency_p50_ms=np.percentile(latencies, 50),\n            latency_p95_ms=np.percentile(latencies, 95),\n            latency_p99_ms=np.percentile(latencies, 99),\n            recall=self._calculate_recall(search_fn, query_vectors, k) if self.ground_truth_fn else 0,\n            qps=total_queries \u002F total_time\n        )\n\n    def _calculate_recall(self, search_fn, queries: np.ndarray, k: int) -> float:\n        \"\"\"Calculate recall against ground truth.\"\"\"\n        if not self.ground_truth_fn:\n            return 0\n\n        correct = 0\n        total = 0\n\n        for query in queries:\n            predicted = set(search_fn(query, k=k))\n            actual = set(self.ground_truth_fn(query, k=k))\n            correct += len(predicted & actual)\n            total += k\n\n        return correct \u002F total\n\n\ndef profile_index_build(\n    build_fn,\n    vectors: np.ndarray,\n    batch_sizes: List[int] = [1000, 10000, 50000]\n) -> dict:\n    \"\"\"Profile index build performance.\"\"\"\n    results = {}\n\n    for batch_size in batch_sizes:\n        times = []\n        for i in range(0, len(vectors), batch_size):\n            batch = vectors[i:i + batch_size]\n            start = time.perf_counter()\n            build_fn(batch)\n            times.append(time.perf_counter() - start)\n\n        results[batch_size] = {\n            \"avg_batch_time_s\": np.mean(times),\n            \"vectors_per_second\": batch_size \u002F np.mean(times)\n        }\n\n    return results\n```\n\n## Best Practices\n\n### Do's\n- **Benchmark with real queries** - Synthetic may not represent production\n- **Monitor recall continuously** - Can degrade with data drift\n- **Start with defaults** - Tune only when needed\n- **Use quantization** - Significant memory savings\n- **Consider tiered storage** - Hot\u002Fcold data separation\n\n### Don'ts\n- **Don't over-optimize early** - Profile first\n- **Don't ignore build time** - Index updates have cost\n- **Don't forget reindexing** - Plan for maintenance\n- **Don't skip warming** - Cold indexes are slow\n\n## Resources\n\n- [HNSW Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1603.09320)\n- [Faiss Wiki](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Ffaiss\u002Fwiki)\n- [ANN Benchmarks](https:\u002F\u002Fann-benchmarks.com\u002F)\n","# Vector Index Tuning\n\nGuide to optimizing vector indexes for production performance.\n\n## When to Use This Skill\n\n- Tuning HNSW parameters\n- Implementing quantization\n- Optimizing memory usage\n- Reducing search latency\n- Balancing recall vs speed\n- Scaling to billions of vectors\n\n## Core Concepts\n\n### 1. Index Type Selection\n\n```\nData Size           Recommended Index\n────────────────────────────────────────\n\u003C 10K vectors  →    Flat (exact search)\n10K - 1M       →    HNSW\n1M - 100M      →    HNSW + Quantization\n> 100M         →    IVF + PQ or DiskANN\n```\n\n### 2. HNSW Parameters\n\n| Parameter | Default | Effect |\n|-----------|---------|--------|\n| **M** | 16 | Connections per node, ↑ = better recall, more memory |\n| **efConstruction** | 100 | Build quality, ↑ = better index, slower build |\n| **efSearch** | 50 | Search quality, ↑ = better recall, slower search |\n\n### 3. Quantization Types\n\n```\nFull Precision (FP32): 4 bytes × dimensions\nHalf Precision (FP16): 2 bytes × dimensions\nINT8 Scalar:           1 byte × dimensions\nProduct Quantization:  ~32-64 bytes total\nBinary:                dimensions\u002F8 bytes\n```\n\n## Templates\n\n### Template 1: HNSW Parameter Tuning\n\n```python\nimport numpy as np\nfrom typing import List, Tuple\nimport time\n\ndef benchmark_hnsw_parameters(\n    vectors: np.ndarray,\n    queries: np.ndarray,\n    ground_truth: np.ndarray,\n    m_values: List[int] = [8, 16, 32, 64],\n    ef_construction_values: List[int] = [64, 128, 256],\n    ef_search_values: List[int] = [32, 64, 128, 256]\n) -> List[dict]:\n    \"\"\"Benchmark different HNSW configurations.\"\"\"\n    import hnswlib\n\n    results = []\n    dim = vectors.shape[1]\n    n = vectors.shape[0]\n\n    for m in m_values:\n        for ef_construction in ef_construction_values:\n            # Build index\n            index = hnswlib.Index(space='cosine', dim=dim)\n            index.init_index(max_elements=n, M=m, ef_construction=ef_construction)\n\n            build_start = time.time()\n            index.add_items(vectors)\n            build_time = time.time() - build_start\n\n            # Get memory usage\n            memory_bytes = index.element_count * (\n                dim * 4 +  # Vector storage\n                m * 2 * 4  # Graph edges (approximate)\n            )\n\n            for ef_search in ef_search_values:\n                index.set_ef(ef_search)\n\n                # Measure search\n                search_start = time.time()\n                labels, distances = index.knn_query(queries, k=10)\n                search_time = time.time() - search_start\n\n                # Calculate recall\n                recall = calculate_recall(labels, ground_truth, k=10)\n\n                results.append({\n                    \"M\": m,\n                    \"ef_construction\": ef_construction,\n                    \"ef_search\": ef_search,\n                    \"build_time_s\": build_time,\n                    \"search_time_ms\": search_time * 1000 \u002F len(queries),\n                    \"recall@10\": recall,\n                    \"memory_mb\": memory_bytes \u002F 1024 \u002F 1024\n                })\n\n    return results\n\n\ndef calculate_recall(predictions: np.ndarray, ground_truth: np.ndarray, k: int) -> float:\n    \"\"\"Calculate recall@k.\"\"\"\n    correct = 0\n    for pred, truth in zip(predictions, ground_truth):\n        correct += len(set(pred[:k]) & set(truth[:k]))\n    return correct \u002F (len(predictions) * k)\n\n\ndef recommend_hnsw_params(\n    num_vectors: int,\n    target_recall: float = 0.95,\n    max_latency_ms: float = 10,\n    available_memory_gb: float = 8\n) -> dict:\n    \"\"\"Recommend HNSW parameters based on requirements.\"\"\"\n\n    # Base recommendations\n    if num_vectors \u003C 100_000:\n        m = 16\n        ef_construction = 100\n    elif num_vectors \u003C 1_000_000:\n        m = 32\n        ef_construction = 200\n    else:\n        m = 48\n        ef_construction = 256\n\n    # Adjust ef_search based on recall target\n    if target_recall >= 0.99:\n        ef_search = 256\n    elif target_recall >= 0.95:\n        ef_search = 128\n    else:\n        ef_search = 64\n\n    return {\n        \"M\": m,\n        \"ef_construction\": ef_construction,\n        \"ef_search\": ef_search,\n        \"notes\": f\"Estimated for {num_vectors:,} vectors, {target_recall:.0%} recall\"\n    }\n```\n\n### Template 2: Quantization Strategies\n\n```python\nimport numpy as np\nfrom typing import Optional\n\nclass VectorQuantizer:\n    \"\"\"Quantization strategies for vector compression.\"\"\"\n\n    @staticmethod\n    def scalar_quantize_int8(\n        vectors: np.ndarray,\n        min_val: Optional[float] = None,\n        max_val: Optional[float] = None\n    ) -> Tuple[np.ndarray, dict]:\n        \"\"\"Scalar quantization to INT8.\"\"\"\n        if min_val is None:\n            min_val = vectors.min()\n        if max_val is None:\n            max_val = vectors.max()\n\n        # Scale to 0-255 range\n        scale = 255.0 \u002F (max_val - min_val)\n        quantized = np.clip(\n            np.round((vectors - min_val) * scale),\n            0, 255\n        ).astype(np.uint8)\n\n        params = {\"min_val\": min_val, \"max_val\": max_val, \"scale\": scale}\n        return quantized, params\n\n    @staticmethod\n    def dequantize_int8(\n        quantized: np.ndarray,\n        params: dict\n    ) -> np.ndarray:\n        \"\"\"Dequantize INT8 vectors.\"\"\"\n        return quantized.astype(np.float32) \u002F params[\"scale\"] + params[\"min_val\"]\n\n    @staticmethod\n    def product_quantize(\n        vectors: np.ndarray,\n        n_subvectors: int = 8,\n        n_centroids: int = 256\n    ) -> Tuple[np.ndarray, dict]:\n        \"\"\"Product quantization for aggressive compression.\"\"\"\n        from sklearn.cluster import KMeans\n\n        n, dim = vectors.shape\n        assert dim % n_subvectors == 0\n        subvector_dim = dim \u002F\u002F n_subvectors\n\n        codebooks = []\n        codes = np.zeros((n, n_subvectors), dtype=np.uint8)\n\n        for i in range(n_subvectors):\n            start = i * subvector_dim\n            end = (i + 1) * subvector_dim\n            subvectors = vectors[:, start:end]\n\n            kmeans = KMeans(n_clusters=n_centroids, random_state=42)\n            codes[:, i] = kmeans.fit_predict(subvectors)\n            codebooks.append(kmeans.cluster_centers_)\n\n        params = {\n            \"codebooks\": codebooks,\n            \"n_subvectors\": n_subvectors,\n            \"subvector_dim\": subvector_dim\n        }\n        return codes, params\n\n    @staticmethod\n    def binary_quantize(vectors: np.ndarray) -> np.ndarray:\n        \"\"\"Binary quantization (sign of each dimension).\"\"\"\n        # Convert to binary: positive = 1, negative = 0\n        binary = (vectors > 0).astype(np.uint8)\n\n        # Pack bits into bytes\n        n, dim = vectors.shape\n        packed_dim = (dim + 7) \u002F\u002F 8\n\n        packed = np.zeros((n, packed_dim), dtype=np.uint8)\n        for i in range(dim):\n            byte_idx = i \u002F\u002F 8\n            bit_idx = i % 8\n            packed[:, byte_idx] |= (binary[:, i] \u003C\u003C bit_idx)\n\n        return packed\n\n\ndef estimate_memory_usage(\n    num_vectors: int,\n    dimensions: int,\n    quantization: str = \"fp32\",\n    index_type: str = \"hnsw\",\n    hnsw_m: int = 16\n) -> dict:\n    \"\"\"Estimate memory usage for different configurations.\"\"\"\n\n    # Vector storage\n    bytes_per_dimension = {\n        \"fp32\": 4,\n        \"fp16\": 2,\n        \"int8\": 1,\n        \"pq\": 0.05,  # Approximate\n        \"binary\": 0.125\n    }\n\n    vector_bytes = num_vectors * dimensions * bytes_per_dimension[quantization]\n\n    # Index overhead\n    if index_type == \"hnsw\":\n        # Each node has ~M*2 edges, each edge is 4 bytes (int32)\n        index_bytes = num_vectors * hnsw_m * 2 * 4\n    elif index_type == \"ivf\":\n        # Inverted lists + centroids\n        index_bytes = num_vectors * 8 + 65536 * dimensions * 4\n    else:\n        index_bytes = 0\n\n    total_bytes = vector_bytes + index_bytes\n\n    return {\n        \"vector_storage_mb\": vector_bytes \u002F 1024 \u002F 1024,\n        \"index_overhead_mb\": index_bytes \u002F 1024 \u002F 1024,\n        \"total_mb\": total_bytes \u002F 1024 \u002F 1024,\n        \"total_gb\": total_bytes \u002F 1024 \u002F 1024 \u002F 1024\n    }\n```\n\n### Template 3: Qdrant Index Configuration\n\n```python\nfrom qdrant_client import QdrantClient\nfrom qdrant_client.http import models\n\ndef create_optimized_collection(\n    client: QdrantClient,\n    collection_name: str,\n    vector_size: int,\n    num_vectors: int,\n    optimize_for: str = \"balanced\"  # \"recall\", \"speed\", \"memory\"\n) -> None:\n    \"\"\"Create collection with optimized settings.\"\"\"\n\n    # HNSW configuration based on optimization target\n    hnsw_configs = {\n        \"recall\": models.HnswConfigDiff(m=32, ef_construct=256),\n        \"speed\": models.HnswConfigDiff(m=16, ef_construct=64),\n        \"balanced\": models.HnswConfigDiff(m=16, ef_construct=128),\n        \"memory\": models.HnswConfigDiff(m=8, ef_construct=64)\n    }\n\n    # Quantization configuration\n    quantization_configs = {\n        \"recall\": None,  # No quantization for max recall\n        \"speed\": models.ScalarQuantization(\n            scalar=models.ScalarQuantizationConfig(\n                type=models.ScalarType.INT8,\n                quantile=0.99,\n                always_ram=True\n            )\n        ),\n        \"balanced\": models.ScalarQuantization(\n            scalar=models.ScalarQuantizationConfig(\n                type=models.ScalarType.INT8,\n                quantile=0.99,\n                always_ram=False\n            )\n        ),\n        \"memory\": models.ProductQuantization(\n            product=models.ProductQuantizationConfig(\n                compression=models.CompressionRatio.X16,\n                always_ram=False\n            )\n        )\n    }\n\n    # Optimizer configuration\n    optimizer_configs = {\n        \"recall\": models.OptimizersConfigDiff(\n            indexing_threshold=10000,\n            memmap_threshold=50000\n        ),\n        \"speed\": models.OptimizersConfigDiff(\n            indexing_threshold=5000,\n            memmap_threshold=20000\n        ),\n        \"balanced\": models.OptimizersConfigDiff(\n            indexing_threshold=20000,\n            memmap_threshold=50000\n        ),\n        \"memory\": models.OptimizersConfigDiff(\n            indexing_threshold=50000,\n            memmap_threshold=10000  # Use disk sooner\n        )\n    }\n\n    client.create_collection(\n        collection_name=collection_name,\n        vectors_config=models.VectorParams(\n            size=vector_size,\n            distance=models.Distance.COSINE\n        ),\n        hnsw_config=hnsw_configs[optimize_for],\n        quantization_config=quantization_configs[optimize_for],\n        optimizers_config=optimizer_configs[optimize_for]\n    )\n\n\ndef tune_search_parameters(\n    client: QdrantClient,\n    collection_name: str,\n    target_recall: float = 0.95\n) -> dict:\n    \"\"\"Tune search parameters for target recall.\"\"\"\n\n    # Search parameter recommendations\n    if target_recall >= 0.99:\n        search_params = models.SearchParams(\n            hnsw_ef=256,\n            exact=False,\n            quantization=models.QuantizationSearchParams(\n                ignore=True,  # Don't use quantization for search\n                rescore=True\n            )\n        )\n    elif target_recall >= 0.95:\n        search_params = models.SearchParams(\n            hnsw_ef=128,\n            exact=False,\n            quantization=models.QuantizationSearchParams(\n                ignore=False,\n                rescore=True,\n                oversampling=2.0\n            )\n        )\n    else:\n        search_params = models.SearchParams(\n            hnsw_ef=64,\n            exact=False,\n            quantization=models.QuantizationSearchParams(\n                ignore=False,\n                rescore=False\n            )\n        )\n\n    return search_params\n```\n\n### Template 4: Performance Monitoring\n\n```python\nimport time\nfrom dataclasses import dataclass\nfrom typing import List\nimport numpy as np\n\n@dataclass\nclass SearchMetrics:\n    latency_p50_ms: float\n    latency_p95_ms: float\n    latency_p99_ms: float\n    recall: float\n    qps: float\n\n\nclass VectorSearchMonitor:\n    \"\"\"Monitor vector search performance.\"\"\"\n\n    def __init__(self, ground_truth_fn=None):\n        self.latencies = []\n        self.recalls = []\n        self.ground_truth_fn = ground_truth_fn\n\n    def measure_search(\n        self,\n        search_fn,\n        query_vectors: np.ndarray,\n        k: int = 10,\n        num_iterations: int = 100\n    ) -> SearchMetrics:\n        \"\"\"Benchmark search performance.\"\"\"\n        latencies = []\n\n        for _ in range(num_iterations):\n            for query in query_vectors:\n                start = time.perf_counter()\n                results = search_fn(query, k=k)\n                latency = (time.perf_counter() - start) * 1000\n                latencies.append(latency)\n\n        latencies = np.array(latencies)\n        total_queries = num_iterations * len(query_vectors)\n        total_time = sum(latencies) \u002F 1000  # seconds\n\n        return SearchMetrics(\n            latency_p50_ms=np.percentile(latencies, 50),\n            latency_p95_ms=np.percentile(latencies, 95),\n            latency_p99_ms=np.percentile(latencies, 99),\n            recall=self._calculate_recall(search_fn, query_vectors, k) if self.ground_truth_fn else 0,\n            qps=total_queries \u002F total_time\n        )\n\n    def _calculate_recall(self, search_fn, queries: np.ndarray, k: int) -> float:\n        \"\"\"Calculate recall against ground truth.\"\"\"\n        if not self.ground_truth_fn:\n            return 0\n\n        correct = 0\n        total = 0\n\n        for query in queries:\n            predicted = set(search_fn(query, k=k))\n            actual = set(self.ground_truth_fn(query, k=k))\n            correct += len(predicted & actual)\n            total += k\n\n        return correct \u002F total\n\n\ndef profile_index_build(\n    build_fn,\n    vectors: np.ndarray,\n    batch_sizes: List[int] = [1000, 10000, 50000]\n) -> dict:\n    \"\"\"Profile index build performance.\"\"\"\n    results = {}\n\n    for batch_size in batch_sizes:\n        times = []\n        for i in range(0, len(vectors), batch_size):\n            batch = vectors[i:i + batch_size]\n            start = time.perf_counter()\n            build_fn(batch)\n            times.append(time.perf_counter() - start)\n\n        results[batch_size] = {\n            \"avg_batch_time_s\": np.mean(times),\n            \"vectors_per_second\": batch_size \u002F np.mean(times)\n        }\n\n    return results\n```\n\n## Best Practices\n\n### Do's\n- **Benchmark with real queries** - Synthetic may not represent production\n- **Monitor recall continuously** - Can degrade with data drift\n- **Start with defaults** - Tune only when needed\n- **Use quantization** - Significant memory savings\n- **Consider tiered storage** - Hot\u002Fcold data separation\n\n### Don'ts\n- **Don't over-optimize early** - Profile first\n- **Don't ignore build time** - Index updates have cost\n- **Don't forget reindexing** - Plan for maintenance\n- **Don't skip warming** - Cold indexes are slow\n\n## Resources\n\n- [HNSW Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F1603.09320)\n- [Faiss Wiki](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Ffaiss\u002Fwiki)\n- [ANN Benchmarks](https:\u002F\u002Fann-benchmarks.com\u002F)\n",{"name":5,"description":7},"main","7850740bd837adb32f0a109ab2af82482196c6f2","skills\u002F_collections\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fvector-index-tuning",1,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills","https:\u002F\u002Fzturnsgo.com","https:\u002F\u002Fgithub.com\u002Fshengdabai","https:\u002F\u002Fzturnsgo.com\u002F","AI × education creator — producer, not consumer. Taught 6,000+ learners; shipping learning tools & daily essays with Claude Code, practicing @xiaolai's way 🥋","2026-11-01 21:28:23",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add shengdabai\u002FTony-Claude-Code-Skills --skill vector-index-tuning",{"tool":59,"command":60},"bunx","bunx skills add shengdabai\u002FTony-Claude-Code-Skills --skill vector-index-tuning",{"tool":62,"command":63},"pnpm","pnpm dlx skills add shengdabai\u002FTony-Claude-Code-Skills --skill vector-index-tuning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fvector-index-tuning\u002FSKILL.md",true,[],[68,77,85,93,101,109],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":72,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":76},"01M73H7XKBHE1ETQDZT94MF9HH","seedance-prompt-en","Write effective prompts for Jimeng Seedance 2.0 multimodal AI video generation. Use when users want to create video prompts using text, images, videos, and audio inputs with the @ reference system. Covers camera movements, effects replication, video extension, editing, music beat-matching, e-commerce ads, short dramas, and educational content.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":81,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":76},"01M73GRE9998Q1DZWC71N41JWG","onepage-pdf","Convert an HTML page into a single continuous-page PDF (one tall page, no pagination breaks), preserving desktop layout, backgrounds and selectable text. Use when the user asks to turn an HTML file\u002Freport\u002Fproposal into a \"single-page PDF\", \"long PDF\", \"不分页 PDF\", \"单页 PDF\", \"长图式 PDF\", or complains that an HTML-to-PDF export breaks into pages or cuts content. Supports optional string redaction with leak verification.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":89,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":76},"01M73GQG1KZDADVPV3MHAT1J5A","native-feel-cross-platform-desktop","Use when the user is designing, prototyping, or rewriting a desktop app that must run on multiple OSes (macOS + Windows, optionally Linux) AND feel indistinguishable from a native app to its users — fast launch, native windowing, native input handling, native materials. Trigger words include \"cross-platform desktop\", \"Electron alternative\", \"Tauri vs native\", \"WebView wrapper\", \"near-native performance\", \"Raycast architecture\", \"WebKit\u002FWebView2 quirks\", \"WKWebView\", \"system tray app\", \"global hotkey app\", \"launcher app\". Do NOT trigger this skill for pure web apps, pure mobile apps, or for greenfield projects that have no native-feel requirement.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":97,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":76},"01M73G87PDNSMKG2FHWCYER6N8","gstack","Router for the gstack skill suite. (gstack)","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":105,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":76},"01M73G3EAQWKCM8DB8D0N0QSVG","fireworks-tech-graph","Use when the user wants to create any technical diagram - architecture, data flow, flowchart, sequence, agent\u002Fmemory, or concept map - and export as SVG+PNG. Trigger on: \"画图\" \"帮我画\" \"生成图\" \"做个图\" \"架构图\" \"流程图\" \"可视化一下\" \"出图\" \"generate diagram\" \"draw diagram\" \"visualize\" or any system\u002Fflow description the user wants illustrated.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":76},"01M73G086AXAH9DW1H5G5N3CPB","dbs-video-extract","接收抖音、小红书、微信视频号等短视频链接或分享文案，查询作品／账号数据并提取语音文字稿，按作者和标题归档为 Markdown。用户希望完整解析短视频、取得数据或文字稿时使用；运行前检查 TikHub 与轻抖使用凭证，缺少时用新手能理解的方式引导购买和安全配置。","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fdbs-video-extract\u002FSKILL.md",[],[]]