AGENT SKILL
big-data
Big data processing and systems
适用职业数据分析师
- License 许可
- MIT
- 最近更新
- 2026-03-01
- Stars
- ★ 3
- Forks
- 2
01 前言 Front Matter
- name
- big-data
- license
- MIT
- metadata
- { "audience": "data-engineers", "category": "data-science" }
- description
- Big data processing and systems
- compatibility
- opencode
03 SKILL.md 正文
What I do
- Design and implement big data architectures
- Process large-scale datasets with distributed systems
- Optimize data pipelines for throughput
- Work with Hadoop, Spark, and cloud data platforms
- Implement real-time streaming data processing
- Manage data lakes and warehouses at scale
When to use me
Use me when:
- Working with datasets beyond single-machine capacity
- Building data pipelines at scale
- Processing streaming data in real-time
- Designing data lake architectures
- Optimizing big data workflows
- Managing petabyte-scale storage
Key Concepts
Big Data Ecosystem
- Storage: HDFS, S3, Azure Blob, GCS
- Processing: Spark, Flink, Hadoop MapReduce
- Query: Hive, Presto, BigQuery, Snowflake
- Stream: Kafka, Pulsar, Kinesis, Dataflow
- Orchestration: Airflow, Dagster, Prefect
Spark DataFrame Example
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum, avg
spark = SparkSession.builder \
.appName("Analytics") \
.config("spark.sql.shuffle.partitions", 200) \
.getOrCreate()
# Read from multiple sources
df = spark.read \
.parquet("s3://data-lake/raw/events/") \
.filter(col("date") >= "2024-01-01")
# Transform and aggregate
result = df.groupBy("user_id", "product_category") \
.agg(
sum("purchase_amount").alias("total_spent"),
avg("purchase_amount").alias("avg_purchase"),
count("*").alias("num_purchases")
) \
.orderBy(col("total_spent").desc())
# Write aggregated results
result.write \
.mode("overwrite") \
.partitionBy("product_category") \
.parquet("s3://data-warehouse/analytics/")
Data Processing Patterns
- Batch: Scheduled, periodic processing
- Streaming: Continuous, real-time
- Lambda: Batch + speed layers
- Kappa: Streaming-only architecture
Cloud Big Data Services
- AWS: EMR, Redshift, Athena, Kinesis
- Azure: Databricks, Synapse, Data Lake
- GCP: Dataproc, BigQuery, Dataflow
04 同仓库更多 Skills
xamarin
Xamarin cross-platform development
★ 3
wrangling
Data wrangling fundamentals
★ 3
windows
Windows operating system administration and scripting
★ 3
wide-column
Wide-column data model
★ 3
wide-column-stores
Wide-column database stores
★ 3
websockets
WebSocket implementation and best practices
★ 3