5. Solution Design and Storage Architecture Detail

09/24/2026 Contributors

Karthikeyan Nagalingam, NetApp

5.1 Design Principles

Configuration over code changes; explicit stage boundaries with typed contracts; consistent artifact routing; fail-fast validation with actionable errors; storage-tier neutrality.

5.2 Stage Design Summary

Stage Design Summary

Stage	Design Summary
Ingestion	`ingestion_tool` selects `s3_direct`, `none`, `airbyte`, or `nifi`; returns normalized metadata
Data Preparation	Discovers/accepts raw StorageGRID tabular keys; normalizes Airbyte/plain CSV; produces deterministic train/val/infer splits; writes a run-stamped prepared-data prefix and manifest to the ONTAP NAS bucket
Data Mobility (XCP)	Copies prepared data from the ONTAP NAS NFS export; `xcp_copy_destination` drives the XCP target and training read path
Model Training	Trains locally or from XCP destination (S3/LustreFS); multi-format (CSV/Parquet/JSON) discovery
Fine-Tuning	Materializes base model from XCP destination; `partial_fit` with expanded class set; republishes tuned model
Inference	Materializes tuned artifacts from same XCP destination; configurable split/text scope
Checkpoint/Resume	`write_stage_checkpoint` persists completion record; `_small_resume_guard` validates/reuses on subsequent runs
Archival	`model_training` archives core trained models immediately; `inferencing` waits for predictions and archives the full result set; optional XCP-input inclusion and raw-folder cleanup

Ingestion

ingestion_tool selects s3_direct, none, airbyte, or nifi; returns normalized metadata

Data Preparation

Discovers/accepts raw StorageGRID tabular keys; normalizes Airbyte/plain CSV; produces deterministic train/val/infer splits; writes a run-stamped prepared-data prefix and manifest to the ONTAP NAS bucket

Data Mobility (XCP)

Copies prepared data from the ONTAP NAS NFS export; xcp_copy_destination drives the XCP target and training read path

Model Training

Trains locally or from XCP destination (S3/LustreFS); multi-format (CSV/Parquet/JSON) discovery

Fine-Tuning

Materializes base model from XCP destination; partial_fit with expanded class set; republishes tuned model

Inference

Materializes tuned artifacts from same XCP destination; configurable split/text scope

Checkpoint/Resume

write_stage_checkpoint persists completion record; _small_resume_guard validates/reuses on subsequent runs

Archival

model_training archives core trained models immediately; inferencing waits for predictions and archives the full result set; optional XCP-input inclusion and raw-folder cleanup

5.3 Configuration-Driven Philosophy

All non-secret storage, engine, and behavioral selections are dag_run.conf parameters — switching between S3 and LustreFS, or Python and Spark, requires no code modification, while Airflow-managed secret storage resolves the required credentials.

5.4 Storage Architecture Detail

The storage architecture separates the AI lifecycle into purpose-built tiers: StorageGRID holds raw and archival objects, the ONTAP NAS bucket holds prepared and run-stamped datasets, and ONTAP S3 or LustreFS provides the selected active training tier. This separation keeps source data, prepared data, model artifacts, and archived evidence independently manageable while preserving lineage through the shared run_stamp.

5.4.1 Storage Layout Diagrams

Tier Logical Storage Path Contents / Artifacts Stored

Tier	Logical Storage Path	Contents / Artifacts Stored
StorageGRID Raw Tier	`s3://raw_bucket/raw_prefix/`	Tabular CSV parts (`tabular_part_*.csv`), text inputs (`text_base.json`, `text_finetune.json`, `text_infer.json`)
ONTAP NAS Prepared Tier	`s3://prepared_bucket/prepared_prefix/run_stamp/`	Formatted splits (`data/tabular_*.csv`), text files, `data_prep_manifest.json`, Lakehouse tables (`tables/` Delta/Iceberg)
Active Training Tier (XCP Dest)	`<xcp_prefix>/formatted/run_stamp/`	Copied data splits, trained model binaries (`artifacts/.bin`), evaluation metrics (`artifacts/_metrics.json`)
StorageGRID Archival Tier	`s3://archive_bucket/archive_prefix/stage/run_stamp/`	Stage-scoped baseline or full model artifacts, prediction evidence (`tabular_predictions.csv`, `text_predictions.json`)
Inter-Tier Lineage & Flow	Raw → Prepared → Active Tier → Archival	End-to-end data movement and artifact lineage linked across all storage tiers via the shared `run_stamp`

StorageGRID Raw Tier

s3://raw_bucket/raw_prefix/

Tabular CSV parts (tabular_part_*.csv), text inputs (text_base.json, text_finetune.json, text_infer.json)

ONTAP NAS Prepared Tier

s3://prepared_bucket/prepared_prefix/run_stamp/

Formatted splits (data/tabular_*.csv), text files, data_prep_manifest.json, Lakehouse tables (tables/ Delta/Iceberg)

Active Training Tier (XCP Dest)

<xcp_prefix>/formatted/run_stamp/

Copied data splits, trained model binaries (artifacts/*.bin), evaluation metrics (artifacts/*_metrics.json)

StorageGRID Archival Tier

s3://archive_bucket/archive_prefix/stage/run_stamp/

Stage-scoped baseline or full model artifacts, prediction evidence (tabular_predictions.csv, text_predictions.json)

Inter-Tier Lineage & Flow

Raw → Prepared → Active Tier → Archival

End-to-end data movement and artifact lineage linked across all storage tiers via the shared run_stamp

This diagram maps the logical bucket and prefix layout for a single pipeline run across all storage tiers:

Logical Storage Layout for a Single Pipeline Run

5.4.2 Storage Sizing Guidance

Size each tier independently according to its role and retention policy. The ONTAP NAS and selected XCP destination must accommodate concurrent active runs, whereas StorageGRID capacity is driven primarily by raw-data and archival retention. Include temporary working capacity and growth headroom when planning peak pipeline concurrency.

Tier	Sizing Basis
StorageGRID raw bucket	Ingestion volume × retention window
ONTAP NAS prepared-data bucket	Prepared dataset size × number of retained runs
XCP destination (S3 or Lustre)	Peak concurrent training dataset size + model artifact overhead
Archival bucket	Retention policy × archive stage selection (model_training = smaller; inferencing = larger)

Tier

Sizing Basis

StorageGRID raw bucket

Ingestion volume × retention window

ONTAP NAS prepared-data bucket

Prepared dataset size × number of retained runs

XCP destination (S3 or Lustre)

Peak concurrent training dataset size + model artifact overhead

Archival bucket

Retention policy × archive stage selection (model_training = smaller; inferencing = larger)

5.4.3 Storage Performance Considerations

The XCP destination is selected per run with xcp_copy_destination, allowing storage performance to match the workload instead of forcing every workload onto one tier. Select LustreFS for high-row-count datasets, many concurrent readers, or I/O-intensive training. Select ONTAP S3 for cost-conscious, elastic-access workloads whose performance requirements do not demand a parallel filesystem. In either case, XCP keeps data and model artifacts aligned with the selected training tier.