[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M70G7E1RY23D8BDPA7JMQ9MR":3,"skill-same:01M70G7E1RY23D8BDPA7JMQ9MR":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":51,"ownerDescription":52,"updatedAt":53,"install":54,"resources":67},"01M70G7E1RY23D8BDPA7JMQ9MR","spark-optimization",null,"Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.","shengdabai\u002FTony-Claude-Code-Skills","shengdabai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F116417419?v=4","01M70F6D2WY5PQV5A92BKYJQEW","Tony Sheng","01M70F6D3GAAAYZ5H4CC1G8VXM",3,"MIT","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"spark optimization","spark","Optimization","jobs","data processing pipelines","data processing","with","use","apache spark","data","tuning","apache","optimize","caching","when","debugging","performance","memory","scaling","processing","2026-11-10 10:12:30",false,"---\nname: spark-optimization\ndescription: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.\n---\n\n# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor | Impact | Solution |\n|--------|--------|----------|\n| **Shuffle** | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew** | Uneven task duration | Salting, broadcast joins |\n| **Serialization** | CPU overhead | Use Kryo, columnar formats |\n| **Memory** | GC pressure, spills | Tune executor memory |\n| **Partitions** | Parallelism | Right-size partitions |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Patterns\n\n### Pattern 1: Optimal Partitioning\n\n```python\n# Calculate optimal partition count\ndef calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:\n    \"\"\"\n    Optimal partition size: 128MB - 256MB\n    Too few: Under-utilization, memory pressure\n    Too many: Task scheduling overhead\n    \"\"\"\n    return max(int(data_size_gb * 1024 \u002F partition_size_mb), 1)\n\n# Repartition for even distribution\ndf_repartitioned = df.repartition(200, \"partition_key\")\n\n# Coalesce to reduce partitions (no shuffle)\ndf_coalesced = df.coalesce(100)\n\n# Partition pruning with predicate pushdown\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .filter(F.col(\"date\") == \"2024-01-01\"))  # Spark pushes this down\n\n# Write with partitioning for future queries\n(df.write\n    .partitionBy(\"year\", \"month\", \"day\")\n    .mode(\"overwrite\")\n    .parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned_output\u002F\"))\n```\n\n### Pattern 2: Join Optimization\n\n```python\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import *\n\n# 1. Broadcast Join - Small table joins\n# Best when: One side \u003C 10MB (configurable)\nsmall_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fsmall_table\u002F\")  # \u003C 10MB\nlarge_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Flarge_table\u002F\")  # TBs\n\n# Explicit broadcast hint\nresult = large_df.join(\n    F.broadcast(small_df),\n    on=\"key\",\n    how=\"left\"\n)\n\n# 2. Sort-Merge Join - Default for large tables\n# Requires shuffle, but handles any size\nresult = large_df1.join(large_df2, on=\"key\", how=\"inner\")\n\n# 3. Bucket Join - Pre-sorted, no shuffle at join time\n# Write bucketed tables\n(df.write\n    .bucketBy(200, \"customer_id\")\n    .sortBy(\"customer_id\")\n    .mode(\"overwrite\")\n    .saveAsTable(\"bucketed_orders\"))\n\n# Join bucketed tables (no shuffle!)\norders = spark.table(\"bucketed_orders\")\ncustomers = spark.table(\"bucketed_customers\")  # Same bucket count\nresult = orders.join(customers, on=\"customer_id\")\n\n# 4. Skew Join Handling\n# Enable AQE skew join optimization\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n\n# Manual salting for severe skew\ndef salt_join(df_skewed, df_other, key_col, num_salts=10):\n    \"\"\"Add salt to distribute skewed keys\"\"\"\n    # Add salt to skewed side\n    df_salted = df_skewed.withColumn(\n        \"salt\",\n        (F.rand() * num_salts).cast(\"int\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Explode other side with all salts\n    df_exploded = df_other.crossJoin(\n        spark.range(num_salts).withColumnRenamed(\"id\", \"salt\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Join on salted key\n    return df_salted.join(df_exploded, on=\"salted_key\", how=\"inner\")\n```\n\n### Pattern 3: Caching and Persistence\n\n```python\nfrom pyspark import StorageLevel\n\n# Cache when reusing DataFrame multiple times\ndf = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\ndf_filtered = df.filter(F.col(\"status\") == \"active\")\n\n# Cache in memory (MEMORY_AND_DISK is default)\ndf_filtered.cache()\n\n# Or with specific storage level\ndf_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)\n\n# Force materialization\ndf_filtered.count()\n\n# Use in multiple actions\nagg1 = df_filtered.groupBy(\"category\").count()\nagg2 = df_filtered.groupBy(\"region\").sum(\"amount\")\n\n# Unpersist when done\ndf_filtered.unpersist()\n\n# Storage levels explained:\n# MEMORY_ONLY - Fast, but may not fit\n# MEMORY_AND_DISK - Spills to disk if needed (recommended)\n# MEMORY_ONLY_SER - Serialized, less memory, more CPU\n# DISK_ONLY - When memory is tight\n# OFF_HEAP - Tungsten off-heap memory\n\n# Checkpoint for complex lineage\nspark.sparkContext.setCheckpointDir(\"s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F\")\ndf_complex = (df\n    .join(other_df, \"key\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\")))\ndf_complex.checkpoint()  # Breaks lineage, materializes\n```\n\n### Pattern 4: Memory Tuning\n\n```python\n# Executor memory configuration\n# spark-submit --executor-memory 8g --executor-cores 4\n\n# Memory breakdown (8GB executor):\n# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)\n#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)\n#   - Remaining 2.4GB for execution (shuffles, joins, sorts)\n# - 40% = 3.2GB for user data structures and internal metadata\n\nspark = (SparkSession.builder\n    .config(\"spark.executor.memory\", \"8g\")\n    .config(\"spark.executor.memoryOverhead\", \"2g\")  # For non-JVM memory\n    .config(\"spark.memory.fraction\", \"0.6\")\n    .config(\"spark.memory.storageFraction\", \"0.5\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    # For memory-intensive operations\n    .config(\"spark.sql.autoBroadcastJoinThreshold\", \"50MB\")\n    # Prevent OOM on large shuffles\n    .config(\"spark.sql.files.maxPartitionBytes\", \"128MB\")\n    .getOrCreate())\n\n# Monitor memory usage\ndef print_memory_usage(spark):\n    \"\"\"Print current memory usage\"\"\"\n    sc = spark.sparkContext\n    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():\n        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)\n        total = mem_status._1() \u002F (1024**3)\n        free = mem_status._2() \u002F (1024**3)\n        print(f\"{executor}: {total:.2f}GB total, {free:.2f}GB free\")\n```\n\n### Pattern 5: Shuffle Optimization\n\n```python\n# Reduce shuffle data size\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"auto\")  # With AQE\nspark.conf.set(\"spark.shuffle.compress\", \"true\")\nspark.conf.set(\"spark.shuffle.spill.compress\", \"true\")\n\n# Pre-aggregate before shuffle\ndf_optimized = (df\n    # Local aggregation first (combiner)\n    .groupBy(\"key\", \"partition_col\")\n    .agg(F.sum(\"value\").alias(\"partial_sum\"))\n    # Then global aggregation\n    .groupBy(\"key\")\n    .agg(F.sum(\"partial_sum\").alias(\"total\")))\n\n# Avoid shuffle with map-side operations\n# BAD: Shuffle for each distinct\ndistinct_count = df.select(\"category\").distinct().count()\n\n# GOOD: Approximate distinct (no shuffle)\napprox_count = df.select(F.approx_count_distinct(\"category\")).collect()[0][0]\n\n# Use coalesce instead of repartition when reducing partitions\ndf_reduced = df.coalesce(10)  # No shuffle\n\n# Optimize shuffle with compression\nspark.conf.set(\"spark.io.compression.codec\", \"lz4\")  # Fast compression\n```\n\n### Pattern 6: Data Format Optimization\n\n```python\n# Parquet optimizations\n(df.write\n    .option(\"compression\", \"snappy\")  # Fast compression\n    .option(\"parquet.block.size\", 128 * 1024 * 1024)  # 128MB row groups\n    .parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\"))\n\n# Column pruning - only read needed columns\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .select(\"id\", \"amount\", \"date\"))  # Spark only reads these columns\n\n# Predicate pushdown - filter at storage level\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned\u002Fyear=2024\u002F\")\n    .filter(F.col(\"status\") == \"active\"))  # Pushed to Parquet reader\n\n# Delta Lake optimizations\n(df.write\n    .format(\"delta\")\n    .option(\"optimizeWrite\", \"true\")  # Bin-packing\n    .option(\"autoCompact\", \"true\")  # Compact small files\n    .mode(\"overwrite\")\n    .save(\"s3:\u002F\u002Fbucket\u002Fdelta_table\u002F\"))\n\n# Z-ordering for multi-dimensional queries\nspark.sql(\"\"\"\n    OPTIMIZE delta.`s3:\u002F\u002Fbucket\u002Fdelta_table\u002F`\n    ZORDER BY (customer_id, date)\n\"\"\")\n```\n\n### Pattern 7: Monitoring and Debugging\n\n```python\n# Enable detailed metrics\nspark.conf.set(\"spark.sql.codegen.wholeStage\", \"true\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n\n# Explain query plan\ndf.explain(mode=\"extended\")\n# Modes: simple, extended, codegen, cost, formatted\n\n# Get physical plan statistics\ndf.explain(mode=\"cost\")\n\n# Monitor task metrics\ndef analyze_stage_metrics(spark):\n    \"\"\"Analyze recent stage metrics\"\"\"\n    status_tracker = spark.sparkContext.statusTracker()\n\n    for stage_id in status_tracker.getActiveStageIds():\n        stage_info = status_tracker.getStageInfo(stage_id)\n        print(f\"Stage {stage_id}:\")\n        print(f\"  Tasks: {stage_info.numTasks}\")\n        print(f\"  Completed: {stage_info.numCompletedTasks}\")\n        print(f\"  Failed: {stage_info.numFailedTasks}\")\n\n# Identify data skew\ndef check_partition_skew(df):\n    \"\"\"Check for partition skew\"\"\"\n    partition_counts = (df\n        .withColumn(\"partition_id\", F.spark_partition_id())\n        .groupBy(\"partition_id\")\n        .count()\n        .orderBy(F.desc(\"count\")))\n\n    partition_counts.show(20)\n\n    stats = partition_counts.select(\n        F.min(\"count\").alias(\"min\"),\n        F.max(\"count\").alias(\"max\"),\n        F.avg(\"count\").alias(\"avg\"),\n        F.stddev(\"count\").alias(\"stddev\")\n    ).collect()[0]\n\n    skew_ratio = stats[\"max\"] \u002F stats[\"avg\"]\n    print(f\"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)\")\n```\n\n## Configuration Cheat Sheet\n\n```python\n# Production configuration template\nspark_configs = {\n    # Adaptive Query Execution (AQE)\n    \"spark.sql.adaptive.enabled\": \"true\",\n    \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n    \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n\n    # Memory\n    \"spark.executor.memory\": \"8g\",\n    \"spark.executor.memoryOverhead\": \"2g\",\n    \"spark.memory.fraction\": \"0.6\",\n    \"spark.memory.storageFraction\": \"0.5\",\n\n    # Parallelism\n    \"spark.sql.shuffle.partitions\": \"200\",\n    \"spark.default.parallelism\": \"200\",\n\n    # Serialization\n    \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n\n    # Compression\n    \"spark.io.compression.codec\": \"lz4\",\n    \"spark.shuffle.compress\": \"true\",\n\n    # Broadcast\n    \"spark.sql.autoBroadcastJoinThreshold\": \"50MB\",\n\n    # File handling\n    \"spark.sql.files.maxPartitionBytes\": \"128MB\",\n    \"spark.sql.files.openCostInBytes\": \"4MB\",\n}\n```\n\n## Best Practices\n\n### Do's\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n\n## Resources\n\n- [Spark Performance Tuning](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html)\n- [Spark Configuration](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fconfiguration.html)\n- [Databricks Optimization Guide](https:\u002F\u002Fdocs.databricks.com\u002Fen\u002Foptimizations\u002Findex.html)\n","# Apache Spark Optimization\n\nProduction patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.\n\n## When to Use This Skill\n\n- Optimizing slow Spark jobs\n- Tuning memory and executor configuration\n- Implementing efficient partitioning strategies\n- Debugging Spark performance issues\n- Scaling Spark pipelines for large datasets\n- Reducing shuffle and data skew\n\n## Core Concepts\n\n### 1. Spark Execution Model\n\n```\nDriver Program\n    ↓\nJob (triggered by action)\n    ↓\nStages (separated by shuffles)\n    ↓\nTasks (one per partition)\n```\n\n### 2. Key Performance Factors\n\n| Factor | Impact | Solution |\n|--------|--------|----------|\n| **Shuffle** | Network I\u002FO, disk I\u002FO | Minimize wide transformations |\n| **Data Skew** | Uneven task duration | Salting, broadcast joins |\n| **Serialization** | CPU overhead | Use Kryo, columnar formats |\n| **Memory** | GC pressure, spills | Tune executor memory |\n| **Partitions** | Parallelism | Right-size partitions |\n\n## Quick Start\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import functions as F\n\n# Create optimized Spark session\nspark = (SparkSession.builder\n    .appName(\"OptimizedJob\")\n    .config(\"spark.sql.adaptive.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\n    .config(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\n    .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    .getOrCreate())\n\n# Read with optimized settings\ndf = (spark.read\n    .format(\"parquet\")\n    .option(\"mergeSchema\", \"false\")\n    .load(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\"))\n\n# Efficient transformations\nresult = (df\n    .filter(F.col(\"date\") >= \"2024-01-01\")\n    .select(\"id\", \"amount\", \"category\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\").alias(\"total\")))\n\nresult.write.mode(\"overwrite\").parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\")\n```\n\n## Patterns\n\n### Pattern 1: Optimal Partitioning\n\n```python\n# Calculate optimal partition count\ndef calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:\n    \"\"\"\n    Optimal partition size: 128MB - 256MB\n    Too few: Under-utilization, memory pressure\n    Too many: Task scheduling overhead\n    \"\"\"\n    return max(int(data_size_gb * 1024 \u002F partition_size_mb), 1)\n\n# Repartition for even distribution\ndf_repartitioned = df.repartition(200, \"partition_key\")\n\n# Coalesce to reduce partitions (no shuffle)\ndf_coalesced = df.coalesce(100)\n\n# Partition pruning with predicate pushdown\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .filter(F.col(\"date\") == \"2024-01-01\"))  # Spark pushes this down\n\n# Write with partitioning for future queries\n(df.write\n    .partitionBy(\"year\", \"month\", \"day\")\n    .mode(\"overwrite\")\n    .parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned_output\u002F\"))\n```\n\n### Pattern 2: Join Optimization\n\n```python\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import *\n\n# 1. Broadcast Join - Small table joins\n# Best when: One side \u003C 10MB (configurable)\nsmall_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fsmall_table\u002F\")  # \u003C 10MB\nlarge_df = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Flarge_table\u002F\")  # TBs\n\n# Explicit broadcast hint\nresult = large_df.join(\n    F.broadcast(small_df),\n    on=\"key\",\n    how=\"left\"\n)\n\n# 2. Sort-Merge Join - Default for large tables\n# Requires shuffle, but handles any size\nresult = large_df1.join(large_df2, on=\"key\", how=\"inner\")\n\n# 3. Bucket Join - Pre-sorted, no shuffle at join time\n# Write bucketed tables\n(df.write\n    .bucketBy(200, \"customer_id\")\n    .sortBy(\"customer_id\")\n    .mode(\"overwrite\")\n    .saveAsTable(\"bucketed_orders\"))\n\n# Join bucketed tables (no shuffle!)\norders = spark.table(\"bucketed_orders\")\ncustomers = spark.table(\"bucketed_customers\")  # Same bucket count\nresult = orders.join(customers, on=\"customer_id\")\n\n# 4. Skew Join Handling\n# Enable AQE skew join optimization\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n\n# Manual salting for severe skew\ndef salt_join(df_skewed, df_other, key_col, num_salts=10):\n    \"\"\"Add salt to distribute skewed keys\"\"\"\n    # Add salt to skewed side\n    df_salted = df_skewed.withColumn(\n        \"salt\",\n        (F.rand() * num_salts).cast(\"int\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Explode other side with all salts\n    df_exploded = df_other.crossJoin(\n        spark.range(num_salts).withColumnRenamed(\"id\", \"salt\")\n    ).withColumn(\n        \"salted_key\",\n        F.concat(F.col(key_col), F.lit(\"_\"), F.col(\"salt\"))\n    )\n\n    # Join on salted key\n    return df_salted.join(df_exploded, on=\"salted_key\", how=\"inner\")\n```\n\n### Pattern 3: Caching and Persistence\n\n```python\nfrom pyspark import StorageLevel\n\n# Cache when reusing DataFrame multiple times\ndf = spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\ndf_filtered = df.filter(F.col(\"status\") == \"active\")\n\n# Cache in memory (MEMORY_AND_DISK is default)\ndf_filtered.cache()\n\n# Or with specific storage level\ndf_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)\n\n# Force materialization\ndf_filtered.count()\n\n# Use in multiple actions\nagg1 = df_filtered.groupBy(\"category\").count()\nagg2 = df_filtered.groupBy(\"region\").sum(\"amount\")\n\n# Unpersist when done\ndf_filtered.unpersist()\n\n# Storage levels explained:\n# MEMORY_ONLY - Fast, but may not fit\n# MEMORY_AND_DISK - Spills to disk if needed (recommended)\n# MEMORY_ONLY_SER - Serialized, less memory, more CPU\n# DISK_ONLY - When memory is tight\n# OFF_HEAP - Tungsten off-heap memory\n\n# Checkpoint for complex lineage\nspark.sparkContext.setCheckpointDir(\"s3:\u002F\u002Fbucket\u002Fcheckpoints\u002F\")\ndf_complex = (df\n    .join(other_df, \"key\")\n    .groupBy(\"category\")\n    .agg(F.sum(\"amount\")))\ndf_complex.checkpoint()  # Breaks lineage, materializes\n```\n\n### Pattern 4: Memory Tuning\n\n```python\n# Executor memory configuration\n# spark-submit --executor-memory 8g --executor-cores 4\n\n# Memory breakdown (8GB executor):\n# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)\n#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)\n#   - Remaining 2.4GB for execution (shuffles, joins, sorts)\n# - 40% = 3.2GB for user data structures and internal metadata\n\nspark = (SparkSession.builder\n    .config(\"spark.executor.memory\", \"8g\")\n    .config(\"spark.executor.memoryOverhead\", \"2g\")  # For non-JVM memory\n    .config(\"spark.memory.fraction\", \"0.6\")\n    .config(\"spark.memory.storageFraction\", \"0.5\")\n    .config(\"spark.sql.shuffle.partitions\", \"200\")\n    # For memory-intensive operations\n    .config(\"spark.sql.autoBroadcastJoinThreshold\", \"50MB\")\n    # Prevent OOM on large shuffles\n    .config(\"spark.sql.files.maxPartitionBytes\", \"128MB\")\n    .getOrCreate())\n\n# Monitor memory usage\ndef print_memory_usage(spark):\n    \"\"\"Print current memory usage\"\"\"\n    sc = spark.sparkContext\n    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():\n        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)\n        total = mem_status._1() \u002F (1024**3)\n        free = mem_status._2() \u002F (1024**3)\n        print(f\"{executor}: {total:.2f}GB total, {free:.2f}GB free\")\n```\n\n### Pattern 5: Shuffle Optimization\n\n```python\n# Reduce shuffle data size\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"auto\")  # With AQE\nspark.conf.set(\"spark.shuffle.compress\", \"true\")\nspark.conf.set(\"spark.shuffle.spill.compress\", \"true\")\n\n# Pre-aggregate before shuffle\ndf_optimized = (df\n    # Local aggregation first (combiner)\n    .groupBy(\"key\", \"partition_col\")\n    .agg(F.sum(\"value\").alias(\"partial_sum\"))\n    # Then global aggregation\n    .groupBy(\"key\")\n    .agg(F.sum(\"partial_sum\").alias(\"total\")))\n\n# Avoid shuffle with map-side operations\n# BAD: Shuffle for each distinct\ndistinct_count = df.select(\"category\").distinct().count()\n\n# GOOD: Approximate distinct (no shuffle)\napprox_count = df.select(F.approx_count_distinct(\"category\")).collect()[0][0]\n\n# Use coalesce instead of repartition when reducing partitions\ndf_reduced = df.coalesce(10)  # No shuffle\n\n# Optimize shuffle with compression\nspark.conf.set(\"spark.io.compression.codec\", \"lz4\")  # Fast compression\n```\n\n### Pattern 6: Data Format Optimization\n\n```python\n# Parquet optimizations\n(df.write\n    .option(\"compression\", \"snappy\")  # Fast compression\n    .option(\"parquet.block.size\", 128 * 1024 * 1024)  # 128MB row groups\n    .parquet(\"s3:\u002F\u002Fbucket\u002Foutput\u002F\"))\n\n# Column pruning - only read needed columns\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fdata\u002F\")\n    .select(\"id\", \"amount\", \"date\"))  # Spark only reads these columns\n\n# Predicate pushdown - filter at storage level\ndf = (spark.read.parquet(\"s3:\u002F\u002Fbucket\u002Fpartitioned\u002Fyear=2024\u002F\")\n    .filter(F.col(\"status\") == \"active\"))  # Pushed to Parquet reader\n\n# Delta Lake optimizations\n(df.write\n    .format(\"delta\")\n    .option(\"optimizeWrite\", \"true\")  # Bin-packing\n    .option(\"autoCompact\", \"true\")  # Compact small files\n    .mode(\"overwrite\")\n    .save(\"s3:\u002F\u002Fbucket\u002Fdelta_table\u002F\"))\n\n# Z-ordering for multi-dimensional queries\nspark.sql(\"\"\"\n    OPTIMIZE delta.`s3:\u002F\u002Fbucket\u002Fdelta_table\u002F`\n    ZORDER BY (customer_id, date)\n\"\"\")\n```\n\n### Pattern 7: Monitoring and Debugging\n\n```python\n# Enable detailed metrics\nspark.conf.set(\"spark.sql.codegen.wholeStage\", \"true\")\nspark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n\n# Explain query plan\ndf.explain(mode=\"extended\")\n# Modes: simple, extended, codegen, cost, formatted\n\n# Get physical plan statistics\ndf.explain(mode=\"cost\")\n\n# Monitor task metrics\ndef analyze_stage_metrics(spark):\n    \"\"\"Analyze recent stage metrics\"\"\"\n    status_tracker = spark.sparkContext.statusTracker()\n\n    for stage_id in status_tracker.getActiveStageIds():\n        stage_info = status_tracker.getStageInfo(stage_id)\n        print(f\"Stage {stage_id}:\")\n        print(f\"  Tasks: {stage_info.numTasks}\")\n        print(f\"  Completed: {stage_info.numCompletedTasks}\")\n        print(f\"  Failed: {stage_info.numFailedTasks}\")\n\n# Identify data skew\ndef check_partition_skew(df):\n    \"\"\"Check for partition skew\"\"\"\n    partition_counts = (df\n        .withColumn(\"partition_id\", F.spark_partition_id())\n        .groupBy(\"partition_id\")\n        .count()\n        .orderBy(F.desc(\"count\")))\n\n    partition_counts.show(20)\n\n    stats = partition_counts.select(\n        F.min(\"count\").alias(\"min\"),\n        F.max(\"count\").alias(\"max\"),\n        F.avg(\"count\").alias(\"avg\"),\n        F.stddev(\"count\").alias(\"stddev\")\n    ).collect()[0]\n\n    skew_ratio = stats[\"max\"] \u002F stats[\"avg\"]\n    print(f\"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)\")\n```\n\n## Configuration Cheat Sheet\n\n```python\n# Production configuration template\nspark_configs = {\n    # Adaptive Query Execution (AQE)\n    \"spark.sql.adaptive.enabled\": \"true\",\n    \"spark.sql.adaptive.coalescePartitions.enabled\": \"true\",\n    \"spark.sql.adaptive.skewJoin.enabled\": \"true\",\n\n    # Memory\n    \"spark.executor.memory\": \"8g\",\n    \"spark.executor.memoryOverhead\": \"2g\",\n    \"spark.memory.fraction\": \"0.6\",\n    \"spark.memory.storageFraction\": \"0.5\",\n\n    # Parallelism\n    \"spark.sql.shuffle.partitions\": \"200\",\n    \"spark.default.parallelism\": \"200\",\n\n    # Serialization\n    \"spark.serializer\": \"org.apache.spark.serializer.KryoSerializer\",\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n\n    # Compression\n    \"spark.io.compression.codec\": \"lz4\",\n    \"spark.shuffle.compress\": \"true\",\n\n    # Broadcast\n    \"spark.sql.autoBroadcastJoinThreshold\": \"50MB\",\n\n    # File handling\n    \"spark.sql.files.maxPartitionBytes\": \"128MB\",\n    \"spark.sql.files.openCostInBytes\": \"4MB\",\n}\n```\n\n## Best Practices\n\n### Do's\n- **Enable AQE** - Adaptive query execution handles many issues\n- **Use Parquet\u002FDelta** - Columnar formats with compression\n- **Broadcast small tables** - Avoid shuffle for small joins\n- **Monitor Spark UI** - Check for skew, spills, GC\n- **Right-size partitions** - 128MB - 256MB per partition\n\n### Don'ts\n- **Don't collect large data** - Keep data distributed\n- **Don't use UDFs unnecessarily** - Use built-in functions\n- **Don't over-cache** - Memory is limited\n- **Don't ignore data skew** - It dominates job time\n- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`\n\n## Resources\n\n- [Spark Performance Tuning](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fsql-performance-tuning.html)\n- [Spark Configuration](https:\u002F\u002Fspark.apache.org\u002Fdocs\u002Flatest\u002Fconfiguration.html)\n- [Databricks Optimization Guide](https:\u002F\u002Fdocs.databricks.com\u002Fen\u002Foptimizations\u002Findex.html)\n",{"name":5,"description":7},"main","7850740bd837adb32f0a109ab2af82482196c6f2","skills\u002F_collections\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization",1,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills","https:\u002F\u002Fzturnsgo.com","https:\u002F\u002Fgithub.com\u002Fshengdabai","https:\u002F\u002Fzturnsgo.com\u002F","AI × education creator — producer, not consumer. Taught 6,000+ learners; shipping learning tools & daily essays with Claude Code, practicing @xiaolai's way 🥋","2026-11-01 21:28:23",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add shengdabai\u002FTony-Claude-Code-Skills --skill spark-optimization",{"tool":60,"command":61},"bunx","bunx skills add shengdabai\u002FTony-Claude-Code-Skills --skill spark-optimization",{"tool":63,"command":64},"pnpm","pnpm dlx skills add shengdabai\u002FTony-Claude-Code-Skills --skill spark-optimization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fspark-optimization\u002FSKILL.md",true,[],[69,78,86,94,102,110],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":73,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":77},"01M73H7XKBHE1ETQDZT94MF9HH","seedance-prompt-en","Write effective prompts for Jimeng Seedance 2.0 multimodal AI video generation. Use when users want to create video prompts using text, images, videos, and audio inputs with the @ reference system. Covers camera movements, effects replication, video extension, editing, music beat-matching, e-commerce ads, short dramas, and educational content.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":82,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":77},"01M73GRE9998Q1DZWC71N41JWG","onepage-pdf","Convert an HTML page into a single continuous-page PDF (one tall page, no pagination breaks), preserving desktop layout, backgrounds and selectable text. Use when the user asks to turn an HTML file\u002Freport\u002Fproposal into a \"single-page PDF\", \"long PDF\", \"不分页 PDF\", \"单页 PDF\", \"长图式 PDF\", or complains that an HTML-to-PDF export breaks into pages or cuts content. Supports optional string redaction with leak verification.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":90,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":77},"01M73GQG1KZDADVPV3MHAT1J5A","native-feel-cross-platform-desktop","Use when the user is designing, prototyping, or rewriting a desktop app that must run on multiple OSes (macOS + Windows, optionally Linux) AND feel indistinguishable from a native app to its users — fast launch, native windowing, native input handling, native materials. Trigger words include \"cross-platform desktop\", \"Electron alternative\", \"Tauri vs native\", \"WebView wrapper\", \"near-native performance\", \"Raycast architecture\", \"WebKit\u002FWebView2 quirks\", \"WKWebView\", \"system tray app\", \"global hotkey app\", \"launcher app\". Do NOT trigger this skill for pure web apps, pure mobile apps, or for greenfield projects that have no native-feel requirement.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":98,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":77},"01M73G87PDNSMKG2FHWCYER6N8","gstack","Router for the gstack skill suite. (gstack)","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":106,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":77},"01M73G3EAQWKCM8DB8D0N0QSVG","fireworks-tech-graph","Use when the user wants to create any technical diagram - architecture, data flow, flowchart, sequence, agent\u002Fmemory, or concept map - and export as SVG+PNG. Trigger on: \"画图\" \"帮我画\" \"生成图\" \"做个图\" \"架构图\" \"流程图\" \"可视化一下\" \"出图\" \"generate diagram\" \"draw diagram\" \"visualize\" or any system\u002Fflow description the user wants illustrated.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":77},"01M73G086AXAH9DW1H5G5N3CPB","dbs-video-extract","接收抖音、小红书、微信视频号等短视频链接或分享文案，查询作品／账号数据并提取语音文字稿，按作者和标题归档为 Markdown。用户希望完整解析短视频、取得数据或文字稿时使用；运行前检查 TikHub 与轻抖使用凭证，缺少时用新手能理解的方式引导购买和安全配置。","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fdbs-video-extract\u002FSKILL.md",[],[]]