parallel-processing
Parallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.
- License 许可
- Apache-2.0
- 最近更新
- 2026-04-21
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- parallel-processing
- description
- Parallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.
03 SKILL.md 正文
Parallel Processing with joblib
Speed up computationally intensive tasks by distributing work across multiple CPU cores.
Basic Usage
from joblib import Parallel, delayed
def process_item(x):
"""Process a single item."""
return x ** 2
# Sequential
results = [process_item(x) for x in range(100)]
# Parallel (uses all available cores)
results = Parallel(n_jobs=-1)(
delayed(process_item)(x) for x in range(100)
)
Key Parameters
- n_jobs:
-1for all cores,1for sequential, or specific number - verbose:
0(silent),10(progress),50(detailed) - backend:
'loky'(CPU-bound, default) or'threading'(I/O-bound)
Grid Search Example
from joblib import Parallel, delayed
from itertools import product
def evaluate_params(param_a, param_b):
"""Evaluate one parameter combination."""
score = expensive_computation(param_a, param_b)
return {'param_a': param_a, 'param_b': param_b, 'score': score}
# Define parameter grid
params = list(product([0.1, 0.5, 1.0], [10, 20, 30]))
# Parallel grid search
results = Parallel(n_jobs=-1, verbose=10)(
delayed(evaluate_params)(a, b) for a, b in params
)
# Filter results
results = [r for r in results if r is not None]
best = max(results, key=lambda x: x['score'])
Pre-computing Shared Data
When all tasks need the same data, pre-compute it once:
# Pre-compute once
shared_data = load_data()
def process_with_shared(params, data):
return compute(params, data)
# Pass shared data to each task
results = Parallel(n_jobs=-1)(
delayed(process_with_shared)(p, shared_data)
for p in param_list
)
Performance Tips
- Only worth it for tasks taking >0.1s per item (overhead cost)
- Watch memory usage - each worker gets a copy of data
- Use
verbose=10to monitor progress
04 同仓库更多 Skills
data-reconciliation
暂无简介
speech-to-text
Transcribe video to timestamped text using Whisper tiny model (pre-installed).
video-processor
Process videos by removing segments and concatenating remaining parts. Use when you need to remove detected pauses/openings from videos, create highlight reels, or batch process segment removals using ffmpeg filter_complex.
silence-detector
Detect initial silence segments in audio/video using energy-based analysis. Use when you need to find low-energy periods at the start of recordings (title slides, setup time, pre-roll silence).
segment-combiner
Combine multiple segment detection results into a unified list. Use when you need to merge segments from different detectors, prepare removal lists for video processing, or consolidate detection outputs.
report-generator
Generate compression reports for video processing. Use when you need to create structured JSON reports with duration statistics, compression ratios, and segment details after video processing.