Trigger a run through trigger_and_wait_ai_pipeline_sklearn.sh with a CONF_JSON payload. The script submits one uniquely named manual run, polls until it reaches a terminal state, and prints local log tails for failed tasks when available. Each primary task emits an effective_config record at startup so operators can verify the evaluated configuration rather than relying on the submitted payload alone.

Monitor DAG state in the Airflow UI or CLI and correlate it with the run ID and run_stamp. The guard logs are grep-friendly: [data_prep] effective_config, [model_training] effective_config, [xcp_copy] effective_config, and [checkpoint] resume_summary. Review these entries to validate the transformation engine, selected XCP destination, training source, and checkpoint reuse decision for each run.

Use this table to connect common task failures to the first corrective action. Resolve source connectivity and mount problems before retrying a run; retrying without correcting an invalid configuration or unavailable endpoint will reproduce the same failure.

Checkpointing applies only to model_training. Set training_checkpoint_reuse_mode=resume_if_exists to reuse a valid completed training result and skip redundant model training, or use verify_only to validate checkpoint eligibility without skipping. Keep reuse disabled (off) during initial testing or whenever training inputs, configuration, or expected artifacts have changed.

Set manual_archive_enabled=true and choose manual_archive_stage deliberately. Use model_training when governance requires the core model baseline as soon as training succeeds; use inferencing when the archive must include final predictions. A run with model_training selected continues to fine-tuning and inference after the archival branch; it simply does not archive their later outputs. Verify uploads under the run-stamped StorageGRID prefix and review [manual_archive] logs for the selected stage, files found, and upload count.

Tune sample_count, spark_driver_memory, spark_executor_memory, and spark_timeout_secs to the input volume and service-level objective. Start with bounded samples to validate data shape and routing, then use sample_count=0 for all-row runs after StorageGRID, ONTAP NAS, XCP, and the selected training tier have sufficient capacity and throughput.

Protect the ONTAP NAS prepared-data bucket with ONTAP Snapshot and backup. Apply StorageGRID protection and retention policies to raw and archival buckets; the archival bucket retains history independent of pipeline re-execution. Combine these storage protections with the training checkpoint record to recover the correct run-scoped data and baseline artifacts after a task, host, or site-level interruption.