Skip to main content
NetApp artificial intelligence solutions

8. Configuration Reference

Contributors nkarthik
PDFs

Karthikeyan Nagalingam, NetApp

The configuration reference describes the runtime parameters that control data ingestion, data preparation, XCP mobility, training, checkpointing, and archival. The ONTAP NAS bucket is used primarily as the prepared-data and XCP staging layer for supportability and portability. In real-time production use, the same prepared data can also be consumed through direct RDMA-capable access paths when that better matches the workload and latency requirements.

All runtime behavior is supplied through dag_run.conf, allowing the same DAG to serve different source systems, transformation engines, storage destinations, recovery policies, and archive scopes. Configure only the parameter groups required for the selected workflow, keep credentials in Airflow Connections or a secrets backend, and use the examples as non-production templates rather than as credential values.

8.1 Ingestion Parameters

Use these settings to select and configure the upstream ingestion mechanism. s3_direct uses the StorageGRID raw-data objects configured in the data-preparation group; Airbyte and NiFi settings are needed only when those respective ingestion tools are selected.

Parameter Default Required Description Example

ingestion_tool

s3_direct

No

Selects ingestion mode

"s3_direct"

airbyte_api_url

None

Airbyte only

Airbyte API endpoint

"http://airbyte:8000"

airbyte_connection_id

None

Airbyte only

Airbyte connection ID

"connection-123"

airbyte_api_token

None

Optional

Bearer token

"<TOKEN>"

airbyte_timeout_secs

30

No

Airbyte timeout

60

nifi_api_url

None

NiFi only

NiFi API endpoint

"http://nifi:8080"

nifi_process_group_id

None

NiFi only

NiFi process group

"abc123"

nifi_timeout_secs

30

No

NiFi timeout

60

8.2 Data Preparation Parameters

These settings control raw-data access, prepared-data output, input discovery, and transformation behavior. The s3_raw_* parameters refer to StorageGRID, while the code-compatible s3_formatted_* parameters identify the ONTAP NAS bucket where prepared, run-stamped data and manifests are written.

Raw StorageGRID / prepared-data ONTAP NAS bucket:

Configure separate endpoints and credentials when StorageGRID and the ONTAP NAS bucket use different S3-compatible services or access policies. The generic fallback keys apply only when a more specific raw or prepared-data setting is absent.

Parameter Default Description Example

s3_raw_bucket

ai-raw-data

StorageGRID raw input bucket

"bucket1"

s3_raw_prefix

example_ai_pipeline/raw

StorageGRID raw-data prefix

"example_ai_pipeline/raw"

s3_raw_endpoint_url

fallback chain

StorageGRID S3 endpoint

"http://10.63.150.62:10444"

s3_raw_access_key_id

fallback chain

Raw access key

"<KEY>"

s3_raw_secret_access_key

fallback chain

Raw secret key

"<SECRET>"

s3_raw_region

aws_region

Raw region

"us-east-1"

s3_formatted_bucket

ai-formatted-data

ONTAP NAS prepared-data output bucket

"prepnasbucket"

s3_formatted_prefix

example_ai_pipeline/formatted

ONTAP NAS prepared-data prefix

"example_ai_pipeline/formatted"

s3_formatted_endpoint_url

fallback chain

ONTAP NAS bucket S3 endpoint

"http://10.63.150.159"

s3_formatted_access_key_id

fallback chain

ONTAP NAS bucket access key

"<KEY>"

s3_formatted_secret_access_key

fallback chain

ONTAP NAS bucket secret key

"<SECRET>"

General fallback: s3_endpoint_url, s3_access_key_id, s3_secret_access_key, s3_session_token, s3_verify, aws_region, aws_access_key_id, aws_secret_access_key, aws_session_token.

Input selection:

Use explicit object-key settings to process known CSV inputs; otherwise, the task discovers eligible tabular objects under the configured StorageGRID raw prefix.

Parameter Default Description Example

s3_tabular_object_keys

Auto-discovery

Explicit list of CSV keys

["raw/demo/a.csv"]

s3_tabular_object_key

None

Single explicit CSV key

"raw/demo/a.csv"

Transformation:

Select the Python path for lightweight local preparation or Spark for distributed preparation. Sampling and the seed apply reproducibly to the generated training, validation, and inference splits.

Parameter Default Description Example

transformation_engine

python

python or spark

"spark"

spark_required

false

Fail instead of fallback

true

table_format

none

none/delta/iceberg

"delta"

table_format_required

false

Fail if format unavailable

true

sample_count

all rows

Row limit (≤0 = all)

14400

seed

7

Reproducible shuffle seed

11

Spark:

These settings apply only to transformation_engine=spark. They control Spark process placement, resource allocation, time limits, and optional Delta Lake or Iceberg runtime dependencies.

Parameter Default Description Example

spark_submit_bin

spark-submit

Spark binary

"/opt/spark/bin/spark-submit"

spark_master

local[*]

Spark master

"local[*]"

spark_driver_memory

4g

Driver memory

"8g"

spark_executor_memory

4g

Executor memory

"8g"

spark_timeout_secs

900/7200

Timeout (0 = disabled)

0

spark_delta_packages

Delta 3.2.0

Delta runtime package

"io.delta:delta-spark_2.12:3.2.0"

spark_iceberg_packages

Iceberg 1.5.2

Iceberg runtime package

"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2"

spark_iceberg_catalog

local

Iceberg catalog name

"local"

8.3 XCP and Training Destination Parameters

Use this group when enable_xcp=true to validate the ONTAP NAS NFS source, invoke the XCP host, and choose the active training tier. xcp_copy_destination is the controlling switch: it selects either the ONTAP S3-specific or LustreFS-specific settings below, and downstream model stages use the same selected tier.

Parameter Default Required Description Example

enable_xcp

false

Yes for XCP

Enables XCP branch

true

xcp_copy_destination

s3

No

s3 or lustrefs

"lustrefs"

xcp_nfs_source_ip

10.63.150.159

XCP preflight

NFS server IP

"10.63.150.159"

xcp_nfs_source_path

/prepnasbucket/example_ai_pipeline

XCP preflight

NFS export path

"/prepnasbucket/example_ai_pipeline"

ssh_user

root

No

XCP host SSH user

"root"

ssh_host

10.63.150.178

No

XCP host address

"10.63.150.178"

S3 destination (required when xcp_copy_destination=s3):

Provide these settings only for the ONTAP S3 training path. The profile or direct credentials allow XCP and the model artifact synchronization logic to access the selected ONTAP S3 bucket and prefix.

Parameter Default Description Example

xcp_s3_endpoint

default

XCP S3 endpoint

"http://10.63.150.161"

xcp_s3_bucket

None

XCP destination bucket

"trainingbucket"

xcp_s3_dest_prefix

example_ai_pipeline

Destination prefix

"example_ai_pipeline"

xcp_s3_training_prefix

xcp_s3_dest_prefix

Training discovery prefix

"example_ai_pipeline"

xcp_s3_profile

ontaps3

Named credential profile

"ontaps3"

xcp_s3_profiles

None

Profile map

{"ontaps3": {…​}}

xcp_s3_access_key_id

None

Direct key fallback

"<KEY>"

xcp_s3_secret_access_key

None

Direct secret fallback

"<SECRET>"

xcp_s3_region

aws_region

XCP S3 region

"us-east-1"

LustreFS destination (required when xcp_copy_destination=lustrefs):

Provide these settings only for the LustreFS training path. The source and destination must be mounted and accessible on the XCP host; the destination must also be accessible to the Airflow worker that runs the model stages.

Parameter Default Description Example

xcp_lustrefs_source_path

/prepnasbucket/example_ai_pipeline

XCP copy source

"/prepnasbucket/example_ai_pipeline"

xcp_lustrefs_dest_path

/mnt/lustre/client

LustreFS mount destination

"/mnt/lustre/client"

xcp_lustrefs_training_subpath

xcp_s3_dest_prefix

Training subdir under mount

"example_ai_pipeline"

xcp_lustrefs_newid

data_prep run stamp

XCP job identifier

"20260901_120000"

Text-source behavior:

These options control how the three text datasets are found after XCP mobility. Explicit keys override discovery; local fallback can be disabled to require that text inputs come from the selected XCP destination.

Parameter Default Description Example

xcp_text_allow_local_fallback

true

Allow local text fallback

false

xcp_text_base_key

Auto

Explicit base-text location

"…​/text_base.json"

xcp_text_finetune_key

Auto

Explicit fine-tune-text location

"…​/text_finetune.json"

xcp_text_infer_key

Auto

Explicit inference-text location

"…​/text_infer.json"

8.4 Model Training Parameters

These settings select the local or XCP-delivered source, limit training volume, and preserve reproducible data ordering. When XCP is enabled, training reads the selected ONTAP S3 or LustreFS tier and publishes the baseline artifacts back to that same destination.

Parameter Default Description Example

enable_xcp

false

Selects XCP or local source

true

xcp_copy_destination

s3

Selects source tier

"lustrefs"

sample_count

all rows

Training sample limit

14400

seed

7

Reproducible shuffle

11

Produces: regression_model.bin, text_vectorizer.bin, text_classifier.bin, train_metrics.json.

8.5 Fine-Tuning Parameters

Fine-tuning materializes the baseline text artifacts from the selected XCP destination, applies incremental learning to the fine-tuning dataset, and republishes the tuned classifier and its metrics. Keep its XCP settings consistent with model training to maintain artifact provenance.

Parameter Default Description Example

enable_xcp

false

Enables artifact materialize/publish

true

xcp_copy_destination

s3

Selects artifact source/destination

"lustrefs"

Consumes: text_vectorizer.bin, text_classifier.bin, text_finetune.json. Produces: text_classifier_tuned.bin, fine_tune_metrics.json.

8.6 Inference Parameters

These flags control the scope of scoring after the tuned model is materialized from the selected training tier. Enable either option when validation or business requirements call for outputs beyond the default inference split and inference text input.

Parameter Default Description Example

infer_all_splits

false

Score train/val/infer splits

true

infer_all_text

false

Score base/finetune/infer text

true

Consumes: regression_model.bin, text_vectorizer.bin, text_classifier_tuned.bin. Produces: tabular_predictions.csv, text_predictions.json.

8.7 Checkpoint Parameters

Checkpointing is scoped to model_training. These settings determine whether the training completion record is created, where it is stored, how upload failures are handled, and whether a later run verifies or reuses a valid completed training result.

Parameter Default Description Example

checkpoint_enabled

true

Enables checkpoint file creation

true

checkpoint_store

local

local or formatted_s3

"formatted_s3"

checkpoint_upload_fail_mode

warn

warn or fail

"fail"

training_checkpoint_reuse_mode

off

off/verify_only/resume_if_exists

"resume_if_exists"

checkpoint_reuse_mode

off

Backward-compatible alias

"verify_only"

8.8 Manual Archive Parameters

Use this group to enable StorageGRID archival, identify the destination bucket and credentials, and choose the archival point. The selected manual_archive_stage controls both the archive timing and whether the archive contains only baseline training artifacts or the complete inference result set.

Parameter Default Required Description Example

manual_archive_enabled

false

No

Enables archival

true

manual_archive_stage

inferencing

No

Archive point and artifact set: model_training for core training artifacts, or inferencing for the full result set

"model_training"

manual_archive_bucket

archivalbucket

Yes when enabled

Archive bucket

"archivalbucket"

manual_archive_prefix

ai-models

No

Archive prefix

"ai-models"

manual_archive_endpoint

default

Custom only

Archive S3 endpoint

"http://10.63.150.62:10444"

manual_archive_profile

sgdlocal

Recommended

Credential profile

"sgdlocal"

manual_archive_profiles

None

Optional

Profile map

{"sgdlocal": {…​}}

manual_archive_include_xcp_inputs

false

No

Include XCP inputs in archive

true

manual_archive_cleanup_raw

false

No

Remove local raw folder post-upload

true

Aliases: fabricpool_archive_* for all manual_archive_* parameters.

8.8.1 Manual Archive Stage Decision

The route_manual_archive_stage branch task reads manual_archive_stage once per DAG run. The selected value determines both when archival runs and which artifacts are uploaded to the StorageGRID archival bucket. Archive objects are stored under s3://<manual_archive_bucket>/<manual_archive_prefix>/<stage>/<run_stamp>/.

manual_archive_stage value Archive timing Artifacts archived Recommended use

model_training

Immediately after model_training; does not wait for fine-tuning or inference

regression_model.bin, text_vectorizer.bin, text_classifier.bin, train_metrics.json

Preserve a reproducible baseline model, minimize archive volume, or retain a checkpoint before later stages

inferencing (default)

After fine_tuning and inferencing complete

All model_training artifacts plus text_classifier_tuned.bin, fine_tune_metrics.json, tabular_predictions.csv, and text_predictions.json

Retain the complete business outcome and prediction evidence for a model run

For either selection, set manual_archive_include_xcp_inputs=true to add locally materialized XCP training inputs when present. Set manual_archive_enabled=false to skip the archival task without changing the training and inference workflow.

8.9 Complete Sample Configuration

This catalog provides starting configurations for the principal deployment and test paths. Every example assumes raw data is in StorageGRID and prepared data is written to the ONTAP NAS bucket. Replace bucket names, endpoint addresses, filesystem paths, and profile names with values from the target environment. Do not place production access keys or secrets in CONF_JSON; configure them through Airflow Connections, Variables, or a secrets backend.

8.9.1 XCP to LustreFS with Spark and Delta Lake

Use this full-throughput example for distributed preparation and high-performance LustreFS model I/O. It processes all eligible tabular files under s3_raw_prefix, writes Delta tables during preparation, persists the training checkpoint to the ONTAP NAS prepared-data bucket, and archives the complete inference result set.

{
  "ingestion_tool": "s3_direct",
  "s3_raw_bucket": "bucket1",
  "s3_raw_prefix": "example_ai_pipeline/raw",
  "s3_formatted_bucket": "prepnasbucket",
  "s3_formatted_prefix": "example_ai_pipeline/formatted",

  "enable_xcp": true,
  "xcp_copy_destination": "lustrefs",
  "xcp_nfs_source_ip": "10.63.150.159",
  "xcp_nfs_source_path": "/prepnasbucket/example_ai_pipeline",
  "xcp_lustrefs_source_path": "/prepnasbucket/example_ai_pipeline",
  "xcp_lustrefs_dest_path": "/mnt/lustre/client",
  "xcp_lustrefs_training_subpath": "example_ai_pipeline",
  "xcp_text_allow_local_fallback": false,

  "transformation_engine": "spark",
  "spark_required": true,
  "table_format": "delta",
  "table_format_required": true,
  "spark_driver_memory": "8g",
  "spark_executor_memory": "8g",
  "spark_timeout_secs": 0,
  "sample_count": 14400,
  "seed": 11,

  "infer_all_splits": true,
  "infer_all_text": true,

  "checkpoint_enabled": true,
  "checkpoint_store": "formatted_s3",
  "training_checkpoint_reuse_mode": "off",

  "manual_archive_enabled": true,
  "manual_archive_stage": "inferencing",
  "manual_archive_bucket": "archivalbucket",
  "manual_archive_prefix": "ai-models",
  "manual_archive_profile": "sgdlocal",
  "manual_archive_cleanup_raw": true
}

8.9.2 No XCP with Python Preparation

Use this baseline configuration for a lightweight functional run. Preparation uses the local Python engine and downstream stages use the pipeline’s local prepared-data and artifact paths; XCP, Spark, table formats, checkpoint reuse, and archival are disabled.

{
    "ingestion_tool": "s3_direct",
    "s3_raw_bucket": "bucket1",
    "s3_raw_prefix": "example_ai_pipeline/raw",
    "s3_formatted_bucket": "prepnasbucket",
    "s3_formatted_prefix": "example_ai_pipeline/formatted",
    "enable_xcp": false,
    "transformation_engine": "python",
    "table_format": "none",
    "sample_count": 200,
    "seed": 11,
    "checkpoint_enabled": false,
    "manual_archive_enabled": false
}

8.9.3 Python Preparation with Two Explicit Input Files

Use this focused test when validating the pipeline against two known StorageGRID CSV objects. s3_tabular_object_keys disables automatic tabular-file discovery; the text inputs still use their expected locations under the raw prefix.

{
    "ingestion_tool": "s3_direct",
    "s3_raw_bucket": "bucket1",
    "s3_raw_prefix": "example_ai_pipeline/raw",
    "s3_tabular_object_keys": [
        "example_ai_pipeline/raw/tabular_part_01.csv",
        "example_ai_pipeline/raw/tabular_part_02.csv"
    ],
    "s3_formatted_bucket": "prepnasbucket",
    "s3_formatted_prefix": "example_ai_pipeline/formatted",
    "enable_xcp": false,
    "transformation_engine": "python",
    "table_format": "none",
    "sample_count": 200,
    "seed": 11,
    "manual_archive_enabled": false
}

8.9.4 Python Preparation with Automatic All-File Discovery

Omit s3_tabular_object_keys and s3_tabular_object_key to process every eligible tabular object under the StorageGRID raw prefix. This example is suitable for an all-files functional or scale test using the Python preparation engine.

{
    "s3_raw_bucket": "bucket1",
    "s3_raw_prefix": "example_ai_pipeline/raw",
    "s3_formatted_bucket": "prepnasbucket",
    "s3_formatted_prefix": "example_ai_pipeline/formatted",
    "enable_xcp": false,
    "transformation_engine": "python",
    "table_format": "none",
    "sample_count": 0,
    "seed": 11,
    "manual_archive_enabled": false
}

sample_count: 0 means that data preparation uses all available rows. Set a positive value for a bounded test run.

8.9.5 XCP to ONTAP S3 with Python Preparation

Use this configuration when prepared data must be copied from the ONTAP NAS NFS export to an ONTAP S3 training bucket. Model training, fine-tuning, and inference materialize and publish artifacts through that same ONTAP S3 destination.

{
    "s3_raw_bucket": "bucket1",
    "s3_raw_prefix": "example_ai_pipeline/raw",
    "s3_formatted_bucket": "prepnasbucket",
    "s3_formatted_prefix": "example_ai_pipeline/formatted",
    "enable_xcp": true,
    "xcp_copy_destination": "s3",
    "xcp_nfs_source_ip": "10.63.150.159",
    "xcp_nfs_source_path": "/prepnasbucket/example_ai_pipeline",
    "xcp_s3_endpoint": "https://ontap-s3.example.com",
    "xcp_s3_bucket": "trainingbucket",
    "xcp_s3_dest_prefix": "example_ai_pipeline",
    "xcp_s3_training_prefix": "example_ai_pipeline",
    "xcp_s3_profile": "ontaps3",
    "transformation_engine": "python",
    "table_format": "none",
    "sample_count": 14400,
    "seed": 11,
    "checkpoint_enabled": true,
    "checkpoint_store": "formatted_s3",
    "manual_archive_enabled": true,
    "manual_archive_stage": "model_training",
    "manual_archive_bucket": "archivalbucket",
    "manual_archive_profile": "storagegrid-archive"
}

8.9.6 XCP to ONTAP S3 with Spark and Iceberg

Use this configuration for distributed preparation with Iceberg table output and ONTAP S3 as the active training destination. Set spark_required and table_format_required to true when a run must fail rather than fall back if Spark or Iceberg is unavailable.

{
    "s3_raw_bucket": "bucket1",
    "s3_raw_prefix": "example_ai_pipeline/raw",
    "s3_formatted_bucket": "prepnasbucket",
    "s3_formatted_prefix": "example_ai_pipeline/formatted",
    "enable_xcp": true,
    "xcp_copy_destination": "s3",
    "xcp_nfs_source_ip": "10.63.150.159",
    "xcp_nfs_source_path": "/prepnasbucket/example_ai_pipeline",
    "xcp_s3_bucket": "trainingbucket",
    "xcp_s3_dest_prefix": "example_ai_pipeline",
    "xcp_s3_profile": "ontaps3",
    "transformation_engine": "spark",
    "spark_required": true,
    "table_format": "iceberg",
    "table_format_required": true,
    "spark_driver_memory": "8g",
    "spark_executor_memory": "8g",
    "spark_timeout_secs": 0,
    "sample_count": 0,
    "seed": 11,
    "manual_archive_enabled": true,
    "manual_archive_stage": "inferencing",
    "manual_archive_bucket": "archivalbucket",
    "manual_archive_profile": "storagegrid-archive"
}

8.9.7 Scenario Selection Summary

Scenario XCP destination Preparation engine Table format Input scope Archive stage

8.9.1

LustreFS

Spark

Delta

All discovered files

inferencing

8.9.2

None

Python

None

All discovered files

Disabled

8.9.3

None

Python

None

Two explicit files

Disabled

8.9.4

None

Python

None

All discovered files

Disabled

8.9.5

ONTAP S3

Python

None

All discovered files

model_training

8.9.6

ONTAP S3

Spark

Iceberg

All discovered files

inferencing

8.9.8 Execution Walkthroughs

Run the examples from the Airflow workspace. The trigger script generates a unique manual__<UTC timestamp> run ID, waits for completion, and returns a nonzero exit code for a failed or timed-out run. The following full configurations use the same CONF_JSON="$(python3 - <<'PY' …​)" pattern as the operational examples. Configure the referenced Airflow-managed connections, XCP profiles, and archive profiles in your secrets backend before running them, and keep CONF_JSON limited to non-secret bucket, endpoint, path, and profile identifiers.

The valid table format name is iceberg. Do not use icerberg, which is not a supported value.

All files, Spark, Iceberg, XCP to ONTAP S3, checkpoint to prepared-data S3, archive after inference:

CONF_JSON="$(python3 - <<'PY'
import json

print(json.dumps({
    "ingestion_tool": "s3_direct",
    "enable_xcp": True,
    "xcp_copy_destination": "s3",
    "xcp_text_allow_local_fallback": True,

    "s3_raw_bucket": "bucket1",
    "s3_raw_prefix": "example_ai_pipeline/raw",
    "s3_raw_endpoint_url": "https://storagegrid.example.com",
    "s3_raw_region": "us-east-1",

    "s3_formatted_bucket": "prepnasbucket",
    "s3_formatted_prefix": "example_ai_pipeline/formatted",
    "s3_formatted_endpoint_url": "https://ontap-nas.example.com",
    "s3_formatted_region": "us-east-1",

    "xcp_nfs_source_ip": "10.63.150.159",
    "xcp_nfs_source_path": "/prepnasbucket/example_ai_pipeline",
    "xcp_s3_endpoint": "https://ontap-s3.example.com",
    "xcp_s3_bucket": "trainingbucket",
    "xcp_s3_dest_prefix": "example_ai_pipeline",
    "xcp_s3_training_prefix": "example_ai_pipeline",
    "xcp_s3_profile": "ontaps3",

    "transformation_engine": "spark",
    "spark_required": True,
    "table_format": "iceberg",
    "table_format_required": True,
    "sample_count": 0,
    "seed": 11,
    "spark_driver_memory": "8g",
    "spark_executor_memory": "8g",
    "spark_timeout_secs": 0,
    "infer_all_splits": True,
    "infer_all_text": True,

    "checkpoint_enabled": True,
    "checkpoint_store": "formatted_s3",
    "manual_archive_enabled": True,
    "manual_archive_stage": "inferencing",
    "manual_archive_bucket": "archivalbucket",
    "manual_archive_prefix": "ai-models",
    "manual_archive_profile": "sgdlocal",
    "manual_archive_endpoint": "https://storagegrid.example.com",
    "manual_archive_cleanup_raw": True,
    "manual_archive_include_xcp_inputs": True,
}, separators=(",", ":")))
PY
)"
CONF_JSON="$CONF_JSON" AIRFLOW_BIN=./.venv/bin/airflow INGESTION_TOOL=s3_direct \
  ./trigger_and_wait_ai_pipeline_sklearn.sh

All files, Spark, Iceberg, XCP to LustreFS, checkpoint to prepared-data S3, archive after inference:

CONF_JSON="$(python3 - <<'PY'
import json

print(json.dumps({
    "ingestion_tool": "s3_direct",
    "enable_xcp": True,
    "xcp_copy_destination": "lustrefs",
    "ssh_user": "root",
    "ssh_host": "10.63.150.178",
    "xcp_text_allow_local_fallback": True,

    "s3_raw_bucket": "bucket1",
    "s3_raw_prefix": "example_ai_pipeline/raw",
    "s3_raw_endpoint_url": "https://storagegrid.example.com",
    "s3_raw_region": "us-east-1",

    "s3_formatted_bucket": "prepnasbucket",
    "s3_formatted_prefix": "example_ai_pipeline/formatted",
    "s3_formatted_endpoint_url": "https://ontap-nas.example.com",
    "s3_formatted_region": "us-east-1",

    "xcp_nfs_source_ip": "10.63.150.159",
    "xcp_nfs_source_path": "/prepnasbucket/example_ai_pipeline",
    "xcp_lustrefs_source_path": "/prepnasbucket/example_ai_pipeline",
    "xcp_lustrefs_dest_path": "/mnt/lustre/client",
    "xcp_lustrefs_training_subpath": "example_ai_pipeline",

    "transformation_engine": "spark",
    "spark_required": True,
    "table_format": "iceberg",
    "table_format_required": True,
    "sample_count": 0,
    "seed": 11,
    "spark_driver_memory": "8g",
    "spark_executor_memory": "8g",
    "spark_timeout_secs": 0,
    "infer_all_splits": True,
    "infer_all_text": True,

    "checkpoint_enabled": True,
    "checkpoint_store": "formatted_s3",
    "manual_archive_enabled": True,
    "manual_archive_stage": "inferencing",
    "manual_archive_bucket": "archivalbucket",
    "manual_archive_prefix": "ai-models",
    "manual_archive_profile": "sgdlocal",
    "manual_archive_endpoint": "https://storagegrid.example.com",
    "manual_archive_cleanup_raw": True,
    "manual_archive_include_xcp_inputs": True,
}, separators=(",", ":")))
PY
)"
AIRFLOW_BIN=./.venv/bin/airflow INGESTION_TOOL=s3_direct \
  ./trigger_and_wait_ai_pipeline_sklearn.sh

For ONTAP S3 mode, pass only the non-secret xcp_s3_profile name in CONF_JSON and define the corresponding credentials on the XCP host or through Airflow-managed secret storage before the run. For LustreFS mode, the xcp_s3_profiles map is intentionally omitted because the training route does not resolve ONTAP S3 credentials. In both examples, define the StorageGRID raw-data, ONTAP NAS prepared-data, and archive credentials through Airflow-managed secret storage rather than inline JSON. Omitting s3_tabular_object_keys and setting sample_count to 0 requests automatic discovery of all eligible tabular files and use of all available rows.

Python, no XCP, two-file test:

CONF_JSON='{"s3_raw_bucket":"bucket1","s3_raw_prefix":"example_ai_pipeline/raw","s3_tabular_object_keys":["example_ai_pipeline/raw/tabular_part_01.csv","example_ai_pipeline/raw/tabular_part_02.csv"],"s3_formatted_bucket":"prepnasbucket","s3_formatted_prefix":"example_ai_pipeline/formatted","enable_xcp":false,"transformation_engine":"python","table_format":"none","sample_count":200,"manual_archive_enabled":false}' \
./trigger_and_wait_ai_pipeline_sklearn.sh

Spark, Delta Lake, XCP to LustreFS:

CONF_JSON='{"enable_xcp":true,"xcp_copy_destination":"lustrefs","xcp_nfs_source_ip":"10.63.150.159","xcp_nfs_source_path":"/prepnasbucket/example_ai_pipeline","xcp_lustrefs_source_path":"/prepnasbucket/example_ai_pipeline","xcp_lustrefs_dest_path":"/mnt/lustre/client","xcp_lustrefs_training_subpath":"example_ai_pipeline","transformation_engine":"spark","spark_required":true,"table_format":"delta","table_format_required":true,"sample_count":14400,"manual_archive_enabled":true,"manual_archive_stage":"inferencing"}' \
./trigger_and_wait_ai_pipeline_sklearn.sh

Spark, Iceberg, XCP to ONTAP S3:

CONF_JSON='{"enable_xcp":true,"xcp_copy_destination":"s3","xcp_nfs_source_ip":"10.63.150.159","xcp_nfs_source_path":"/prepnasbucket/example_ai_pipeline","xcp_s3_bucket":"trainingbucket","xcp_s3_dest_prefix":"example_ai_pipeline","xcp_s3_profile":"ontaps3","transformation_engine":"spark","spark_required":true,"table_format":"iceberg","table_format_required":true,"sample_count":0,"manual_archive_enabled":true,"manual_archive_stage":"inferencing"}' \
./trigger_and_wait_ai_pipeline_sklearn.sh

Before an XCP run, verify the configured ONTAP NAS NFS path and destination accessibility from the XCP host. After a successful run, review task logs for [data_prep] effective_config, [xcp_copy] effective_config, [model_training] effective_config, and [manual_archive] to confirm the selected engine, input scope, destination, and archive stage.