spatial-raw-processing

spatial-raw-processing

Use from a step

This is CLI_ONLY: ST-Pipeline consumes FASTQ files, barcode coordinates and reference indexes. It is not an in-memory analysis function.

from skills._sdk.notebook import run_cli

output = run_cli("spatial-raw-processing", "--input", "data/run_bundle.json",
                 inputs=["data/run_bundle.json", "data/R1.fastq.gz",
                         "data/R2.fastq.gz", "data/barcodes.tsv"])

Record the reference index version in the module README. The executable example uses synthetic upstream outputs and does not run ST-Pipeline.

When to use

The user has paired-end spatial-transcriptomics FASTQ files (read1 = spatial barcode + UMI, read2 = cDNA) plus a STAR genome index, and wants the standard ST-Pipeline run that produces a raw_counts.h5ad with one row per spatial spot. Single backend: st_pipeline (calls run_stpipeline from skills/spatial/_lib/stpipeline_adapter.py).

After this skill, chain to spatial-preprocess for QC + normalisation. For non-spatial scRNA FASTQ use sc-fastq-qc. For bulk RNA-seq read QC use bulkrna-read-qc.

Inputs & Outputs

Inputs

• Input kinds: file, directory
• Modalities: visium, slideseq
• File types: .fastq, .fq, .json, .yaml, .yml
• FASTQ structure: valid first record; paired layout
• Directory layouts (any): paired-fastq

Outputs

• tables/gene_qc.csv
• tables/raw_gene_qc.csv
• tables/raw_processing_run_summary.csv
• tables/raw_processing_spatial_points.csv
• tables/raw_spot_qc.csv
• tables/raw_top_genes.csv
• tables/run_summary.csv
• tables/saturation_curve.csv
• tables/spatial_coordinates.csv
• tables/spot_qc.csv
• tables/stage_summary.csv
• tables/top_genes.csv
• figures/raw_detected_genes_spatial.png
• figures/raw_spot_qc_histograms.png
• figures/raw_top_genes_barplot.png
• figures/raw_total_counts_spatial.png
• figures/st_pipeline_saturation_curve.png
• figures/st_pipeline_stage_attrition.png
• omicsclaw_stpipeline_run.json
• raw_counts.h5ad
• st_pipeline.stderr.txt
• st_pipeline.stdout.txt
• report.md
• result.json
• Processed AnnData (saves_h5ad) — adds obs: barcode, x_array, y_array; obsm: spatial

Flow

1. Parse args (or load bundle JSON / YAML from positional --input).
2. _apply_effective_defaults fills missing parameter values (threads, trimming, UMI ranges, etc.).
3. _validate_real_run_bundle: check read1 / read2 / ids / ref-map exist and are well-typed; reject duplicate read1=read2; verify FASTQ extension.
4. Call run_stpipeline(...) which shells out to ST-Pipeline (requires the stpipeline binary on PATH or --stpipeline-repo + --bin-path).
5. Wrap the resulting count matrix into AnnData with X = raw_counts, layers["counts"], raw = raw_counts_snapshot.
6. Save raw_counts.h5ad and result.json. Print "next: spatial-preprocess on raw_counts.h5ad".

Gotchas

• All input failures raise typed exceptions wrapped in SystemExit(1). spatial_raw_processing.py catches DataError / DependencyError / ParameterError / ProcessingError and re-raises as SystemExit(1). The originating raises live in _validate_real_run_bundle — it raises ParameterError(f"Missing required parameter: {key}") for missing read1/read2/ids; raises DataError(...) for non-existent files; raises DataError("Resolved read1/read2 inputs must be FASTQ files.") for non-FASTQ extensions; raises ParameterError for read1==read2; raises DataError for missing / wrong-type STAR index dir; raises DataError only when --ref-annotation was provided but the path is missing or not a file (the param itself is optional — omitting it doesn't raise).
• --read1 / --read2 / --ids / --ref-map are all required for real runs (not enforced by argparse required=True, validated later). Missing any → ParameterError. Demo mode skips this validation entirely.
• The output filename is always raw_counts.h5ad (spatial_raw_processing.py). It's not configurable — the contract is consumed by spatial-preprocess. Multiple runs to the same --output will overwrite.
• QC tables and figures are written. tables/spot_qc.csv and figures/raw_total_counts_spatial.png describe the count matrix; upstream stage and saturation plots depend on available pipeline metrics.
• Demo mode skips ST-Pipeline entirely. spatial_raw_processing.py calls create_demo_upstream_outputs(...) to fabricate a synthetic raw_counts.h5ad. Useful for plumbing checks; does NOT exercise the FASTQ → matrix code path.
• --platform is a metadata label only. _build_parser documents it as "Label recorded in outputs"; ST-Pipeline doesn't branch on it. Common values: visium, visium_hd, slideseq, custom strings.

Key CLI

# Demo (synthetic raw_counts.h5ad — does NOT run ST-Pipeline)
python skills/spatial/spatial-raw-processing/spatial_raw_processing.py --demo --output /tmp/spatial_raw_demo

# Real run with explicit args
python skills/spatial/spatial-raw-processing/spatial_raw_processing.py \
  --read1 sample_R1.fastq.gz --read2 sample_R2.fastq.gz \
  --ids barcodes.tsv \
  --ref-map /refs/star_index_human \
  --ref-annotation /refs/genes.gtf \
  --exp-name visium_001 --platform visium \
  --threads 16 \
  --output results/

# Real run from bundle JSON
python skills/spatial/spatial-raw-processing/spatial_raw_processing.py \
  --input run_bundle.json --output results/

# Slide-seq with custom UMI range
python skills/spatial/spatial-raw-processing/spatial_raw_processing.py \
  --read1 R1.fq.gz --read2 R2.fq.gz --ids barcodes.tsv \
  --ref-map /refs/star_index --platform slideseq \
  --umi-start-position 1 --umi-end-position 8 \
  --output results/

See also

• references/parameters.md — every CLI flag, ST-Pipeline option mapping
• references/methodology.md — when ST-Pipeline wins vs Space Ranger; barcode-ID format
• references/output_contract.md — raw_counts.h5ad schema
• Adjacent skills: spatial-preprocess (downstream — required next step; consumes raw_counts.h5ad), bulkrna-read-qc / sc-fastq-qc (parallel — non-spatial FASTQ paths)

Dependencies

Python packages this skill's script needs. They are not installed for you — check before a long run.

anndata, matplotlib, numpy, pandas, PyYAML, scipy, seaborn