Skip to main content
NetApp Solutions

Test procedure

Contributors kevin-hoke

This section describes the tasks needed to complete the validation.

Prerequisites

To execute the tasks outlined in this section, you must have access to a Linux or macOS host with the following tools installed and configured:

  • Kubectl (configured for access to an existing Kubernetes cluster)

    • Installation and configuration instructions can be found here.

  • NetApp DataOps Toolkit for Kubernetes

    • Installation instructions can be found here.

Scenario 1 – On-demand inferencing in JupyterLab

  1. Create a Kubernetes namespace for AI/ML inferencing workloads.

    $ kubectl create namespace inference
    namespace/inference created
  2. Use the NetApp DataOps Toolkit to provision a persistent volume for storing the data on which you will perform the inferencing.

    $ netapp_dataops_k8s_cli.py create volume --namespace=inference --pvc-name=inference-data --size=50Gi
    Creating PersistentVolumeClaim (PVC) 'inference-data' in namespace 'inference'.
    PersistentVolumeClaim (PVC) 'inference-data' created. Waiting for Kubernetes to bind volume to PVC.
    Volume successfully created and bound to PersistentVolumeClaim (PVC) 'inference-data' in namespace 'inference'.
  3. Use the NetApp DataOps Toolkit to create a new JupyterLab workspace. Mount the persistent volume that was created in the previous step by using the --mount- pvc option. Allocate NVIDIA GPUs to the workspace as necessary by using the -- nvidia-gpu option.

    In the following example, the persistent volume inference-data is mounted to the JupyterLab workspace container at /home/jovyan/data. When using official Project Jupyter container images, /home/jovyan is presented as the top-level directory within the JupyterLab web interface.

    $ netapp_dataops_k8s_cli.py create jupyterlab --namespace=inference --workspace-name=live-inference --size=50Gi --nvidia-gpu=2 --mount-pvc=inference-data:/home/jovyan/data
    Set workspace password (this password will be required in order to access the workspace):
    Re-enter password:
    Creating persistent volume for workspace...
    Creating PersistentVolumeClaim (PVC) 'ntap-dsutil-jupyterlab-live-inference' in namespace 'inference'.
    PersistentVolumeClaim (PVC) 'ntap-dsutil-jupyterlab-live-inference' created. Waiting for Kubernetes to bind volume to PVC.
    Volume successfully created and bound to PersistentVolumeClaim (PVC) 'ntap-dsutil-jupyterlab-live-inference' in namespace 'inference'.
    Creating Service 'ntap-dsutil-jupyterlab-live-inference' in namespace 'inference'.
    Service successfully created.
    Attaching Additional PVC: 'inference-data' at mount_path: '/home/jovyan/data'.
    Creating Deployment 'ntap-dsutil-jupyterlab-live-inference' in namespace 'inference'.
    Deployment 'ntap-dsutil-jupyterlab-live-inference' created.
    Waiting for Deployment 'ntap-dsutil-jupyterlab-live-inference' to reach Ready state.
    Deployment successfully created.
    Workspace successfully created.
    To access workspace, navigate to http://192.168.0.152:32721
  4. Access the JupyterLab workspace by using the URL specified in the output of the create jupyterlab command. The data directory represents the persistent volume that was mounted to the workspace.

    ai protopia image3
  5. Open the data directory and upload the files on which the inferencing is to be performed. When files are uploaded to the data directory, they are automatically stored on the persistent volume that was mounted to the workspace. To upload files, click the Upload Files icon, as shown in the following image.

    ai protopia image4
  6. Return to the top-level directory and create a new notebook.

    ai protopia image5
  7. Add inferencing code to the notebook. The following example shows inferencing code for an image detection use case.

    ai protopia image6
    ai protopia image7
  8. Add Protopia obfuscation to your inferencing code. Protopia works directly with customers to provide use-case specific documentation and is outside of the scope of this technical report. The following example shows inferencing code for an image detection use case with Protopia obfuscation added.

    ai protopia image8
    ai protopia image9

Scenario 2 – Batch inferencing on Kubernetes

  1. Create a Kubernetes namespace for AI/ML inferencing workloads.

    $ kubectl create namespace inference
    namespace/inference created
  2. Use the NetApp DataOps Toolkit to provision a persistent volume for storing the data on which you will perform the inferencing.

    $ netapp_dataops_k8s_cli.py create volume --namespace=inference --pvc-name=inference-data --size=50Gi
    Creating PersistentVolumeClaim (PVC) 'inference-data' in namespace 'inference'.
    PersistentVolumeClaim (PVC) 'inference-data' created. Waiting for Kubernetes to bind volume to PVC.
    Volume successfully created and bound to PersistentVolumeClaim (PVC) 'inference-data' in namespace 'inference'.
  3. Populate the new persistent volume with the data on which you will perform the inferencing.

    There are several methods for loading data onto a PVC. If your data is currently stored in an S3-compatible object storage platform, such as NetApp StorageGRID or Amazon S3, then you can use NetApp DataOps Toolkit S3 Data Mover capabilities. Another simple method is to create a JupyterLab workspace and then upload files through the JupyterLab web interface, as outlined in Steps 3 to 5 in the section “Scenario 1 – On-demand inferencing in JupyterLab.”

  4. Create a Kubernetes job for your batch inferencing task. The following example shows a batch inferencing job for an image detection use case. This job performs inferencing on each image in a set of images and writes inferencing accuracy metrics to stdout.

    $ vi inference-job-raw.yaml
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: netapp-inference-raw
      namespace: inference
    spec:
      backoffLimit: 5
      template:
        spec:
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: inference-data
          - name: dshm
            emptyDir:
              medium: Memory
          containers:
          - name: inference
            image:: netapp-protopia-inference:latest
            imagePullPolicy: IfNotPresent
            command: ["python3", "run-accuracy-measurement.py", "--dataset", "/data/netapp-face-detection/FDDB"]
            resources:
              limits:
                nvidia.com/gpu: 2
            volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /dev/shm
              name: dshm
          restartPolicy: Never
    $ kubectl create -f inference-job-raw.yaml
    job.batch/netapp-inference-raw created
  5. Confirm that the inferencing job completed successfully.

    $ kubectl -n inference logs netapp-inference-raw-255sp
    100%|██████████| 89/89 [00:52<00:00,  1.68it/s]
    Reading Predictions : 100%|██████████| 10/10 [00:01<00:00,  6.23it/s]
    Predicting ... : 100%|██████████| 10/10 [00:16<00:00,  1.64s/it]
    ==================== Results ====================
    FDDB-fold-1 Val AP: 0.9491256561145955
    FDDB-fold-2 Val AP: 0.9205024466101926
    FDDB-fold-3 Val AP: 0.9253013871078468
    FDDB-fold-4 Val AP: 0.9399781485863011
    FDDB-fold-5 Val AP: 0.9504280149478732
    FDDB-fold-6 Val AP: 0.9416473519339292
    FDDB-fold-7 Val AP: 0.9241631566241117
    FDDB-fold-8 Val AP: 0.9072663297546659
    FDDB-fold-9 Val AP: 0.9339648715035469
    FDDB-fold-10 Val AP: 0.9447707905560152
    FDDB Dataset Average AP: 0.9337148153739079
    =================================================
    mAP: 0.9337148153739079
  6. Add Protopia obfuscation to your inferencing job. You can find use case-specific instructions for adding Protopia obfuscation directly from Protopia, which is outside of the scope of this technical report. The following example shows a batch inferencing job for a face detection use case with Protopia obfuscation added by using an ALPHA value of 0.8. This job applies Protopia obfuscation before performing inferencing for each image in a set of images and then writes inferencing accuracy metrics to stdout.

    We repeated this step for ALPHA values 0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 0.9, and 0.95. You can see the results in “Inferencing accuracy comparison.”

    $ vi inference-job-protopia-0.8.yaml
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: netapp-inference-protopia-0.8
      namespace: inference
    spec:
      backoffLimit: 5
      template:
        spec:
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: inference-data
          - name: dshm
            emptyDir:
              medium: Memory
          containers:
          - name: inference
            image:: netapp-protopia-inference:latest
            imagePullPolicy: IfNotPresent
            env:
            - name: ALPHA
              value: "0.8"
            command: ["python3", "run-accuracy-measurement.py", "--dataset", "/data/netapp-face-detection/FDDB", "--alpha", "$(ALPHA)", "--noisy"]
            resources:
              limits:
                nvidia.com/gpu: 2
            volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /dev/shm
              name: dshm
          restartPolicy: Never
    $ kubectl create -f inference-job-protopia-0.8.yaml
    job.batch/netapp-inference-protopia-0.8 created
  7. Confirm that the inferencing job completed successfully.

    $ kubectl -n inference logs netapp-inference-protopia-0.8-b4dkz
    100%|██████████| 89/89 [01:05<00:00,  1.37it/s]
    Reading Predictions : 100%|██████████| 10/10 [00:02<00:00,  3.67it/s]
    Predicting ... : 100%|██████████| 10/10 [00:22<00:00,  2.24s/it]
    ==================== Results ====================
    FDDB-fold-1 Val AP: 0.8953066115834589
    FDDB-fold-2 Val AP: 0.8819580264029936
    FDDB-fold-3 Val AP: 0.8781107458462862
    FDDB-fold-4 Val AP: 0.9085731346308461
    FDDB-fold-5 Val AP: 0.9166445508275378
    FDDB-fold-6 Val AP: 0.9101178994188819
    FDDB-fold-7 Val AP: 0.8383443678423771
    FDDB-fold-8 Val AP: 0.8476311547659464
    FDDB-fold-9 Val AP: 0.8739624502111121
    FDDB-fold-10 Val AP: 0.8905468076424851
    FDDB Dataset Average AP: 0.8841195749171925
    =================================================
    mAP: 0.8841195749171925

Scenario 3 – NVIDIA Triton Inference Server

  1. Create a Kubernetes namespace for AI/ML inferencing workloads.

    $ kubectl create namespace inference
    namespace/inference created
  2. Use the NetApp DataOps Toolkit to provision a persistent volume to use as a model repository for the NVIDIA Triton Inference Server.

    $ netapp_dataops_k8s_cli.py create volume --namespace=inference --pvc-name=triton-model-repo --size=100Gi
    Creating PersistentVolumeClaim (PVC) 'triton-model-repo' in namespace 'inference'.
    PersistentVolumeClaim (PVC) 'triton-model-repo' created. Waiting for Kubernetes to bind volume to PVC.
    Volume successfully created and bound to PersistentVolumeClaim (PVC) 'triton-model-repo' in namespace 'inference'.
  3. Store your model on the new persistent volume in a format that is recognized by the NVIDIA Triton Inference Server.

    There are several methods for loading data onto a PVC. A simple method is to create a JupyterLab workspace and then upload files through the JupyterLab web interface, as outlined in steps 3 to 5 in “Scenario 1 – On-demand inferencing in JupyterLab. ”

  4. Use NetApp DataOps Toolkit to deploy a new NVIDIA Triton Inference Server instance.

    $ netapp_dataops_k8s_cli.py create triton-server --namespace=inference --server-name=netapp-inference --model-repo-pvc-name=triton-model-repo
    Creating Service 'ntap-dsutil-triton-netapp-inference' in namespace 'inference'.
    Service successfully created.
    Creating Deployment 'ntap-dsutil-triton-netapp-inference' in namespace 'inference'.
    Deployment 'ntap-dsutil-triton-netapp-inference' created.
    Waiting for Deployment 'ntap-dsutil-triton-netapp-inference' to reach Ready state.
    Deployment successfully created.
    Server successfully created.
    Server endpoints:
    http: 192.168.0.152: 31208
    grpc: 192.168.0.152: 32736
    metrics: 192.168.0.152: 30009/metrics
  5. Use a Triton client SDK to perform an inferencing task. The following Python code excerpt uses the Triton Python client SDK to perform an inferencing task for an face detection use case. This example calls the Triton API and passes in an image for inferencing. The Triton Inference Server then receives the request, invokes the model, and returns the inferencing output as part of the API results.

    # get current frame
    frame = input_image
    # preprocess input
    preprocessed_input = preprocess_input(frame)
    preprocessed_input = torch.Tensor(preprocessed_input).to(device)
    # run forward pass
    clean_activation = clean_model_head(preprocessed_input)  # runs the first few layers
    ######################################################################################
    #          pass clean image to Triton Inference Server API for inferencing           #
    ######################################################################################
    triton_client = httpclient.InferenceServerClient(url="192.168.0.152:31208", verbose=False)
    model_name = "face_detection_base"
    inputs = []
    outputs = []
    inputs.append(httpclient.InferInput("INPUT__0", [1, 128, 32, 32], "FP32"))
    inputs[0].set_data_from_numpy(clean_activation.detach().cpu().numpy(), binary_data=False)
    outputs.append(httpclient.InferRequestedOutput("OUTPUT__0", binary_data=False))
    outputs.append(httpclient.InferRequestedOutput("OUTPUT__1", binary_data=False))
    results = triton_client.infer(
        model_name,
        inputs,
        outputs=outputs,
        #query_params=query_params,
        headers=None,
        request_compression_algorithm=None,
        response_compression_algorithm=None)
    #print(results.get_response())
    statistics = triton_client.get_inference_statistics(model_name=model_name, headers=None)
    print(statistics)
    if len(statistics["model_stats"]) != 1:
        print("FAILED: Inference Statistics")
        sys.exit(1)
    
    loc_numpy = results.as_numpy("OUTPUT__0")
    pred_numpy = results.as_numpy("OUTPUT__1")
    ######################################################################################
    # postprocess output
    clean_pred = (loc_numpy, pred_numpy)
    clean_outputs = postprocess_outputs(
        clean_pred, [[input_image_width, input_image_height]], priors, THRESHOLD
    )
    # draw rectangles
    clean_frame = copy.deepcopy(frame)  # needs to be deep copy
    for (x1, y1, x2, y2, s) in clean_outputs[0]:
        x1, y1 = int(x1), int(y1)
        x2, y2 = int(x2), int(y2)
        cv2.rectangle(clean_frame, (x1, y1), (x2, y2), (0, 0, 255), 4)
  6. Add Protopia obfuscation to your inferencing code. You can find use case-specific instructions for adding Protopia obfuscation directly from Protopia; however, this process is outside the scope of this technical report. The following example shows the same Python code that is shown in the preceding step 5, but with Protopia obfuscation added.

    Note that the Protopia obfuscation is applied to the image before it is passed to the Triton API. Thus, the non-obfuscated image never leaves the local machine. Only the obfuscated image is passed across the network. This workflow is applicable to use cases in which data is collected within a trusted zone but then needs to be passed outside of that trusted zone for inferencing. Without Protopia obfuscation, it is not possible to implement this type of workflow without sensitive data ever leaving the trusted zone.

    # get current frame
    frame = input_image
    # preprocess input
    preprocessed_input = preprocess_input(frame)
    preprocessed_input = torch.Tensor(preprocessed_input).to(device)
    # run forward pass
    not_noisy_activation = noisy_model_head(preprocessed_input)  # runs the first few layers
    ##################################################################
    #          obfuscate image locally prior to inferencing          #
    #          SINGLE ADITIONAL LINE FOR PRIVATE INFERENCE           #
    ##################################################################
    noisy_activation = noisy_model_noise(not_noisy_activation)
    ##################################################################
    ###########################################################################################
    #          pass obfuscated image to Triton Inference Server API for inferencing           #
    ###########################################################################################
    triton_client = httpclient.InferenceServerClient(url="192.168.0.152:31208", verbose=False)
    model_name = "face_detection_noisy"
    inputs = []
    outputs = []
    inputs.append(httpclient.InferInput("INPUT__0", [1, 128, 32, 32], "FP32"))
    inputs[0].set_data_from_numpy(noisy_activation.detach().cpu().numpy(), binary_data=False)
    outputs.append(httpclient.InferRequestedOutput("OUTPUT__0", binary_data=False))
    outputs.append(httpclient.InferRequestedOutput("OUTPUT__1", binary_data=False))
    results = triton_client.infer(
        model_name,
        inputs,
        outputs=outputs,
        #query_params=query_params,
        headers=None,
        request_compression_algorithm=None,
        response_compression_algorithm=None)
    #print(results.get_response())
    statistics = triton_client.get_inference_statistics(model_name=model_name, headers=None)
    print(statistics)
    if len(statistics["model_stats"]) != 1:
        print("FAILED: Inference Statistics")
        sys.exit(1)
    
    loc_numpy = results.as_numpy("OUTPUT__0")
    pred_numpy = results.as_numpy("OUTPUT__1")
    ###########################################################################################
    
    # postprocess output
    noisy_pred = (loc_numpy, pred_numpy)
    noisy_outputs = postprocess_outputs(
        noisy_pred, [[input_image_width, input_image_height]], priors, THRESHOLD * 0.5
    )
    # get reconstruction of the noisy activation
    noisy_reconstruction = decoder_function(noisy_activation)
    noisy_reconstruction = noisy_reconstruction.detach().cpu().numpy()[0]
    noisy_reconstruction = unpreprocess_output(
        noisy_reconstruction, (input_image_width, input_image_height), True
    ).astype(np.uint8)
    # draw rectangles
    for (x1, y1, x2, y2, s) in noisy_outputs[0]:
        x1, y1 = int(x1), int(y1)
        x2, y2 = int(x2), int(y2)
        cv2.rectangle(noisy_reconstruction, (x1, y1), (x2, y2), (0, 0, 255), 4)