name: 'radiomics-pathomics-fusion-agent' description: 'AI-powered multimodal fusion of radiology (CT/MRI/PET) and pathology (H&E/IHC) imaging with clinical and genomic data for comprehensive cancer diagnostics and treatment prediction.' measurable_outcome: Execute skill workflow successfully with valid output within 15 minutes. allowed-tools:

• read_file
• run_shell_command

Radiomics Pathomics Fusion Agent

The Radiomics Pathomics Fusion Agent integrates multimodal medical imaging data from radiology (CT, MRI, PET) and digital pathology (H&E, IHC whole slide images) with clinical and genomic data using deep learning fusion architectures. It enables comprehensive cancer phenotyping, treatment response prediction, and prognostic modeling.

When to Use This Skill

• When predicting treatment response using multimodal imaging.
• For comprehensive tumor phenotyping combining macro and micro views.
• To identify imaging biomarkers correlated with genomic features.
• When building prognostic models from combined radiology-pathology.
• For AI-powered second opinion integrating all imaging modalities.

Core Capabilities

1. Cross-Modal Fusion: Integrate radiology and pathology features using attention.

2. Radiomics Extraction: Compute 3D texture, shape, intensity features from CT/MRI.

3. Pathomics Extraction: Extract histopathological features from WSI.

4. Clinical Integration: Combine imaging with clinical variables and genomics.

5. Treatment Response Prediction: Predict chemotherapy, immunotherapy response.

6. Survival Prediction: Multi-modal prognostic modeling.

Supported Imaging Modalities

Fusion Architectures

Workflow

1. Input: CT/MRI DICOM, pathology WSI, clinical data, optional genomics.

2. Segmentation: Tumor ROI extraction from radiology.

3. Radiomics: Extract 3D radiomic features.

4. Pathomics: Extract histopathology features via foundation models.

5. Fusion: Multimodal feature integration.

6. Prediction: Treatment response, survival, biomarker prediction.

7. Output: Integrated predictions, attention maps, explanations.

Example Usage

User: "Predict immunotherapy response for this lung cancer patient using their CT scan and biopsy pathology."

Agent Action:

python3 Skills/Oncology/Radiomics_Pathomics_Fusion_Agent/fusion_predict.py \
    --ct_dicom ct_scan/ \
    --wsi_path biopsy.svs \
    --clinical_data patient_clinical.json \
    --genomic_data tumor_wes.vcf \
    --task immunotherapy_response \
    --cancer_type nsclc \
    --fusion_method attention \
    --output fusion_prediction/

Radiomic Feature Categories

Pathomics Feature Categories

Output Components

Clinical Applications

AI/ML Components

Radiomics Pipeline:

• PyRadiomics for feature extraction
• 3D-CNN for learned features
• Transformer for volumetric analysis

Pathomics Pipeline:

• Foundation models (CONCH, UNI, TITAN)
• MIL (Multiple Instance Learning) for WSI
• Graph networks for spatial patterns

Fusion Models:

• Cross-attention transformers
• Multimodal variational autoencoders
• Contrastive learning for alignment

Prerequisites

• Python 3.10+
• PyRadiomics, SimpleITK
• OpenSlide, HistoEncoder
• PyTorch, transformers
• CONCH/TITAN model weights
• GPU with 16GB+ VRAM

Related Skills

• Pathology_AI/CONCH_Agent - Pathology foundation model
• Radiology_AI agents - Modality-specific analysis
• Pan_Cancer_MultiOmics_Agent - Genomic integration
• TMB_Estimation_Agent - Tumor mutational burden

Multimodal Integration Strategies

Special Considerations

1. Data Alignment: Ensure imaging from same timepoint
2. Missing Modalities: Handle incomplete multimodal data
3. Class Imbalance: Balance training across outcomes
4. Interpretability: Attention maps for clinical trust
5. Validation: External multi-site validation essential

Quality Control

Regulatory Considerations

Author

AI Group - Biomedical AI Platform