Workflow

IMPERANDI passes a cohort table from one stage to the next. Image data remains on disk; CSV path columns preserve the link between each source volume and its derived artifacts.

DICOM roots / archives
        │
        ▼
 parse ─────► dicom_index.csv
        │
        ▼
 clean ─────► dicom_index_clean.csv
        │
        ▼
 convert ───► nifti_index.csv + NIfTI images
        │
        ├────────► phase ─────► totalseg_* metadata
        │
        ▼
 segment ───► mask_* paths
        │
        ▼
 radiomics ─► feature columns

Parse

parse discovers files below one or more roots, reads the default selected DICOM tags plus any supplied with --tags, and resolves patient_key, study_id, and series_id.

ID source modes are:

  • auto (default): prefer configured DICOM tags and fall back to path parts.

  • tags: derive IDs from DICOM tags.

  • path: derive IDs from the expected patient/study/series directory layout.

ZIP, TAR, TAR.GZ, and TGZ inputs can be read without manually unpacking the whole dataset. Archive recursion is bounded by --archive_max_depth; temporary materialization is removed unless --keep_archive_cache is set.

Clean

clean groups instances into volumes, standardizes dates and times, orders exams/acquisitions, and rejects unsuitable data. The implemented filters cover non-CT data, localizers and secondary images, non-axial geometry, noise/body region patterns, implausible volume length, pixel spacing, and slice thickness. Missing geometry is generally retained for later review rather than silently treated as a failure.

The default accepted reconstructed length is 30–1700 mm. Override it with --volume-length-min-mm and --volume-length-max-mm when a protocol calls for different bounds.

Convert

convert materializes each curated series and delegates DICOM-to-NIfTI conversion to dicom2nifti. It works in parallel and records per-volume failures separately. Its input needs a dicom_path representation produced by ingest; output rows receive nifti_path.

Segment, phase, and radiomics

segment runs manifest-defined TotalSegmentator tasks and optional mask post-processing. phase runs TotalSegmentator’s CT contrast-phase extractor. radiomics computes PyRadiomics features for every mask_* column, including an organ-minus-tumor strategy when paired organ and tumor masks are present.

Phase can run immediately after conversion. Segmentation must precede radiomics because radiomics requires one or more mask columns.

Checkpoints and resume

parse, convert, segment, phase, and radiomics checkpoint long runs. Resume is enabled by default when the saved command state and input fingerprint match. Common controls are:

  • --checkpoint_every_rows N: flush after N processed rows.

  • --checkpoint_every_sec T: flush after T seconds.

  • --no_resume: ignore matching checkpoint state and start a fresh run.

  • --strict_resume: hash input contents instead of relying on the lightweight fingerprint; this is safer but slower on large inputs.

Changing material arguments or inputs invalidates an incompatible checkpoint. Do not manually edit checkpoint/state files while a command is running.

Example Slurm batch script

For scheduled runs, a single Slurm job can execute the full pipeline with explicit paths between stages:

#!/bin/bash
#SBATCH --job-name=imperandi-pipeline
#SBATCH --partition=compute
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=24:00:00
#SBATCH --output=%x-%j.out

set -euo pipefail

# Activate the environment where IMPERANDI is installed.
source /path/to/venv/bin/activate

PROJECT_ROOT=/path/to/project
DICOM_ROOT=/path/to/dicom
TABLE_DIR="$PROJECT_ROOT/tables"
NIFTI_DIR="$PROJECT_ROOT/nifti"
MANIFEST="$PROJECT_ROOT/site-a.json"
WORKERS="${SLURM_CPUS_PER_TASK:-4}"

mkdir -p "$TABLE_DIR" "$NIFTI_DIR"

imperandi ingest \
  --root_path "$DICOM_ROOT" \
  --output_dir "$TABLE_DIR" \
  --manifest "$MANIFEST"

imperandi convert \
  --csv_path "$TABLE_DIR/dicom_index_clean.csv" \
  --output_dir "$NIFTI_DIR" \
  --csv_path_out "$TABLE_DIR/nifti_index.csv" \
  --num_workers "$WORKERS"

imperandi segment \
  --csv_path "$TABLE_DIR/nifti_index.csv" \
  --csv_path_out "$TABLE_DIR/nifti_index_segmented.csv" \
  --manifest "$MANIFEST" \
  --num_workers "$WORKERS"

imperandi phase \
  --csv_path "$TABLE_DIR/nifti_index_segmented.csv" \
  --csv_path_out "$TABLE_DIR/nifti_index_phased.csv"

imperandi radiomics \
  --csv_path "$TABLE_DIR/nifti_index_phased.csv" \
  --csv_path_out "$TABLE_DIR/nifti_index_radiomics.csv" \
  --manifest "$MANIFEST"

Adjust Slurm resources for your cohort size and manifest complexity. On re-submission, matching checkpoints let long stages continue instead of starting from scratch unless you pass --no_resume.

Operational recommendations

  • Keep raw DICOM roots read-only.

  • Store each stage’s CSV under versioned or run-specific paths.

  • Start with a small cohort and --num_workers 1 when validating a manifest.

  • Preserve error CSVs and logs with the corresponding output table.

  • Use explicit output paths in automation instead of relying on defaults.