[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CWGVM3TB087GE5X5F3ZXYE":3,"skill-same:01M6CWGVM3TB087GE5X5F3ZXYE":56},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":28,"linkOnly":29,"markdownContent":30,"markdownBody":31,"frontMatter":32,"refName":36,"commitSha":37,"skillDirectory":5,"forksCount":38,"repositoryHtmlUrl":39,"repositoryHomepageUrl":6,"ownerHtmlUrl":40,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":41,"install":42,"resources":55},"01M6CWGVM3TB087GE5X5F3ZXYE","big-data",null,"Big data processing and systems","ffsshhttiikk\u002Fopencode-agents-skills","ffsshhttiikk","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F226346827?v=4","01M6CWD9RR3Y7VRQX3MXYC8FYH","01M6CWD9WVPYAZW0J82G1VKGHY",3,"MIT","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fbig-data\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[23,24,25,26,27],"big data","data","science","data processing","opencode","2026-11-10 10:12:30",false,"---\nname: big-data\ndescription: Big data processing and systems\nlicense: MIT\ncompatibility: opencode\nmetadata:\n  audience: data-engineers\n  category: data-science\n---\n\n## What I do\n\n- Design and implement big data architectures\n- Process large-scale datasets with distributed systems\n- Optimize data pipelines for throughput\n- Work with Hadoop, Spark, and cloud data platforms\n- Implement real-time streaming data processing\n- Manage data lakes and warehouses at scale\n\n## When to use me\n\nUse me when:\n- Working with datasets beyond single-machine capacity\n- Building data pipelines at scale\n- Processing streaming data in real-time\n- Designing data lake architectures\n- Optimizing big data workflows\n- Managing petabyte-scale storage\n\n## Key Concepts\n\n### Big Data Ecosystem\n- **Storage**: HDFS, S3, Azure Blob, GCS\n- **Processing**: Spark, Flink, Hadoop MapReduce\n- **Query**: Hive, Presto, BigQuery, Snowflake\n- **Stream**: Kafka, Pulsar, Kinesis, Dataflow\n- **Orchestration**: Airflow, Dagster, Prefect\n\n### Spark DataFrame Example\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import col, sum, avg\n\nspark = SparkSession.builder \\\n    .appName(\"Analytics\") \\\n    .config(\"spark.sql.shuffle.partitions\", 200) \\\n    .getOrCreate()\n\n# Read from multiple sources\ndf = spark.read \\\n    .parquet(\"s3:\u002F\u002Fdata-lake\u002Fraw\u002Fevents\u002F\") \\\n    .filter(col(\"date\") >= \"2024-01-01\")\n\n# Transform and aggregate\nresult = df.groupBy(\"user_id\", \"product_category\") \\\n    .agg(\n        sum(\"purchase_amount\").alias(\"total_spent\"),\n        avg(\"purchase_amount\").alias(\"avg_purchase\"),\n        count(\"*\").alias(\"num_purchases\")\n    ) \\\n    .orderBy(col(\"total_spent\").desc())\n\n# Write aggregated results\nresult.write \\\n    .mode(\"overwrite\") \\\n    .partitionBy(\"product_category\") \\\n    .parquet(\"s3:\u002F\u002Fdata-warehouse\u002Fanalytics\u002F\")\n```\n\n### Data Processing Patterns\n- **Batch**: Scheduled, periodic processing\n- **Streaming**: Continuous, real-time\n- **Lambda**: Batch + speed layers\n- **Kappa**: Streaming-only architecture\n\n### Cloud Big Data Services\n- **AWS**: EMR, Redshift, Athena, Kinesis\n- **Azure**: Databricks, Synapse, Data Lake\n- **GCP**: Dataproc, BigQuery, Dataflow\n","## What I do\n\n- Design and implement big data architectures\n- Process large-scale datasets with distributed systems\n- Optimize data pipelines for throughput\n- Work with Hadoop, Spark, and cloud data platforms\n- Implement real-time streaming data processing\n- Manage data lakes and warehouses at scale\n\n## When to use me\n\nUse me when:\n- Working with datasets beyond single-machine capacity\n- Building data pipelines at scale\n- Processing streaming data in real-time\n- Designing data lake architectures\n- Optimizing big data workflows\n- Managing petabyte-scale storage\n\n## Key Concepts\n\n### Big Data Ecosystem\n- **Storage**: HDFS, S3, Azure Blob, GCS\n- **Processing**: Spark, Flink, Hadoop MapReduce\n- **Query**: Hive, Presto, BigQuery, Snowflake\n- **Stream**: Kafka, Pulsar, Kinesis, Dataflow\n- **Orchestration**: Airflow, Dagster, Prefect\n\n### Spark DataFrame Example\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import col, sum, avg\n\nspark = SparkSession.builder \\\n    .appName(\"Analytics\") \\\n    .config(\"spark.sql.shuffle.partitions\", 200) \\\n    .getOrCreate()\n\n# Read from multiple sources\ndf = spark.read \\\n    .parquet(\"s3:\u002F\u002Fdata-lake\u002Fraw\u002Fevents\u002F\") \\\n    .filter(col(\"date\") >= \"2024-01-01\")\n\n# Transform and aggregate\nresult = df.groupBy(\"user_id\", \"product_category\") \\\n    .agg(\n        sum(\"purchase_amount\").alias(\"total_spent\"),\n        avg(\"purchase_amount\").alias(\"avg_purchase\"),\n        count(\"*\").alias(\"num_purchases\")\n    ) \\\n    .orderBy(col(\"total_spent\").desc())\n\n# Write aggregated results\nresult.write \\\n    .mode(\"overwrite\") \\\n    .partitionBy(\"product_category\") \\\n    .parquet(\"s3:\u002F\u002Fdata-warehouse\u002Fanalytics\u002F\")\n```\n\n### Data Processing Patterns\n- **Batch**: Scheduled, periodic processing\n- **Streaming**: Continuous, real-time\n- **Lambda**: Batch + speed layers\n- **Kappa**: Streaming-only architecture\n\n### Cloud Big Data Services\n- **AWS**: EMR, Redshift, Athena, Kinesis\n- **Azure**: Databricks, Synapse, Data Lake\n- **GCP**: Dataproc, BigQuery, Dataflow\n",{"name":5,"license":14,"metadata":33,"description":7,"compatibility":27},{"audience":34,"category":35},"data-engineers","data-science","main","5f7a3e453c98261f93d19a6a7de44ad8f7b72d41",2,"https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk","2026-03-01 06:55:40",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":54},[44,47,50],{"tool":45,"command":46},"npx","npx skills add ffsshhttiikk\u002Fopencode-agents-skills --skill big-data",{"tool":48,"command":49},"bunx","bunx skills add ffsshhttiikk\u002Fopencode-agents-skills --skill big-data",{"tool":51,"command":52},"pnpm","pnpm dlx skills add ffsshhttiikk\u002Fopencode-agents-skills --skill big-data","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fbig-data\u002FSKILL.md",true,[],[57,68,76,85,95,104],{"skillNo":58,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":63,"publishedAt":28},"01M6CXT9TXDF79BWJTMQAPB49G","xamarin","Xamarin cross-platform development","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fxamarin\u002FSKILL.md",[],[59,64,65,66,27,67],"platform","cross-platform","mobile","development",{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":28},"01M6CXT8AD04MFYDA32VW3FKAG","wrangling","Data wrangling fundamentals","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fwrangling\u002FSKILL.md",[],[70,75,27,25,24],"data wrangling",{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":28},"01M6CXT717GCN7SRRNPBDP4508","windows","Windows operating system administration and scripting","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fwindows\u002FSKILL.md",[],[78,83,27,84],"system","scripting",{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":28},"01M6CXT54K794NMEM3XAPZZDKT","wide-column","Wide-column data model","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fwide-column\u002FSKILL.md",[],[92,93,27,94,24],"wide column","databases","model",{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":28},"01M6CXT28FMT2P4V9GZE67WA8E","wide-column-stores","Wide-column database stores","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fwide-column-stores\u002FSKILL.md",[],[102,93,27,103],"wide column stores","database",{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":28},"01M6CXT0D0M0Q4T2N7C2RX7DSN","websockets","WebSocket implementation and best practices","https:\u002F\u002Fgithub.com\u002Fffsshhttiikk\u002Fopencode-agents-skills\u002Fblob\u002Fmain\u002Fwebsockets\u002FSKILL.md",[],[106,111,112,113,27,114,115],"websocket","real-time","time","best practices","implementation"]