Configure Working Environment
Suggest changes
PDFs
Copy the Notebook set_env-Example.ipynb as set_env.ipynb. Open and edit set_env.ipynb. This notebook sets variables for credentials, file locations, and execution drivers.
If you follow the instructions above, the following steps are the only changes to make:
-
Obtain this value from the Iguazio services dashboard:
docker_registryExample:
docker-registry.default-tenant.app.clusterq.iguaziodev.com:80 -
Change
adminto your Iguazio username:IGZ_CONTAINER_PATH = '/users/admin'The following are the ONTAP system connection details. Include the volume name that was generated when Trident was installed. The following setting is for an on-premises ONTAP cluster:
ontapClusterMgmtHostname = '0.0.0.0' ontapClusterAdminUsername = 'USER' ontapClusterAdminPassword = 'PASSWORD' sourceVolumeName = 'SOURCE VOLUME'
The following setting is for Cloud Volumes ONTAP:
MANAGER=ontapClusterMgmtHostname svm='svm' email='email' password=ontapClusterAdminPassword weid="weid" volume=sourceVolumeName
Create Base Docker Images
Everything you need to build an ML pipeline is included in the Iguazio platform. The developer can define the specifications of the Docker images required to run the pipeline and execute the image creation from Jupyter Notebook. Open the notebook create- images.ipynb and Run All Cells.
This notebook creates two images that we use in the pipeline.
-
iguazio/netapp.Used to handle ML tasks.
-
netapp/pipeline. Contains utilities to handle NetApp Snapshot copies.
Review Individual Jupyter Notebooks
The following table lists the libraries and frameworks we used to build this task. All these components have been fully integrated with Iguazio’s role- based access and security controls.
| Libraries/Framework | Description |
|---|---|
MLRun |
An managed by Iguazio to enable the assembly, execution, and monitoring of an ML/AI pipeline. |
Nuclio |
A serverless functions framework integrated with Iguazio. Also available as an open-source project managed by Iguazio. |
Kubeflow |
A Kubernetes-based framework to deploy the pipeline. This is also an open-source project to which Iguazio contributes. It is integrated with Iguazio for added security and integration with the rest of the infrastructure. |
Docker |
A Docker registry run as a service in the Iguazio platform. You can also change this to connect to your registry. |
NetApp Cloud Volumes |
Cloud Volumes running on AWS give us access to large amounts of data and the ability to take Snapshot copies to version the datasets used for training. |
Trident |
Trident is an open-source project managed by NetApp. It facilitates the integration with storage and compute resources in Kubernetes. |
We used several notebooks to construct the ML pipeline. Each notebook can be tested individually before being brought together in the pipeline. We cover each notebook individually following the deployment flow of this demonstration application.
The desired result is a pipeline that trains a model based on a Snapshot copy of the data and deploys the model for inference. A block diagram of a completed MLRun pipeline is shown in the following image.
Deploy Data Generation Function
This section describes how we used Nuclio serverless functions to generate network device data. The use case is adapted from an Iguazio client that deployed the pipeline and used Iguazio services to monitor and predict network device failures.
We simulated data coming from network devices. Executing the Jupyter notebook data- generator.ipynb creates a serverless function that runs every 10 minutes and generates a Parquet file with new data. To deploy the function, run all the cells in this notebook. See the Nuclio website to review any unfamiliar components in this notebook.
A cell with the following comment is ignored when generating the function. Every cell in the notebook is assumed to be part of the function. Import the Nuclio module to enable %nuclio magic.
# nuclio: ignore import nuclio
In the spec for the function, we defined the environment in which the function executes, how it is triggered, and the resources it consumes.
spec = nuclio.ConfigSpec(config={"spec.triggers.inference.kind":"cron",
"spec.triggers.inference.attributes.interval" :"10m",
"spec.readinessTimeoutSeconds" : 60,
"spec.minReplicas" : 1},……
The init_context function is invoked by the Nuclio framework upon initialization of the function.
def init_context(context):
….
Any code not in a function is invoked when the function initializes. When you invoke it, a handler function is executed. You can change the name of the handler and specify it in the function spec.
def handler(context, event):
…
You can test the function from the notebook prior to deployment.
%%time # nuclio: ignore init_context(context) event = nuclio.Event(body='') output = handler(context, event) output
The function can be deployed from the notebook or it can be deployed from a CI/CD pipeline (adapting this code).
addr = nuclio.deploy_file(name='generator',project='netops',spec=spec, tag='v1.1')
Pipeline Notebooks
These notebooks are not meant to be executed individually for this setup. This is just a review of each notebook. We invoked them as part of the pipeline. To execute them individually, review the MLRun documentation to execute them as Kubernetes jobs.
snap_cv.ipynb
This notebook handles the Cloud Volume Snapshot copies at the beginning of the pipeline. It passes the name of the volume to the pipeline context. This notebook invokes a shell script to handle the Snapshot copy. While running in the pipeline, the execution context contains variables to help locate all files needed for execution. While writing this code, the developer does not have to worry about the file location in the container that executes it. As described later, this application is deployed with all its dependencies, and it is the definition of the pipeline parameters that provides the execution context.
command = os.path.join(context.get_param('APP_DIR'),"snap_cv.sh")
The created Snapshot copy location is placed in the MLRun context to be consumed by steps in the pipeline.
context.log_result('snapVolumeDetails',snap_path)
The next three notebooks are run in parallel.
data-prep.ipynb
Raw metrics must be turned into features to enable model training. This notebook reads the raw metrics from the Snapshot directory and writes the features for model training to the NetApp volume.
When running in the context of the pipeline, the input DATA_DIR contains the Snapshot copy location.
metrics_table = os.path.join(str(mlruncontext.get_input('DATA_DIR', os.getenv('DATA_DIR','/netpp'))),
mlruncontext.get_param('metrics_table', os.getenv('metrics_table','netops_metrics_parquet')))
describe.ipynb
To visualize the incoming metrics, we deploy a pipeline step that provides plots and graphs that are available through the Kubeflow and MLRun UIs. Each execution has its own version of this visualization tool.
ax.set_title("features correlation")
plt.savefig(os.path.join(base_path, "plots/corr.png"))
context.log_artifact(PlotArtifact("correlation", body=plt.gcf()), local_path="plots/corr.html")
deploy-feature-function.ipynb
We continuously monitor the metrics looking for anomalies. This notebook creates a serverless function that generates the features need to run prediction on incoming metrics. This notebook invokes the creation of the function. The function code is in the notebook data- prep.ipynb. Notice that we use the same notebook as a step in the pipeline for this purpose.
training.ipynb
After we create the features, we trigger the model training. The output of this step is the model to be used for inferencing. We also collect statistics to keep track of each execution (experiment).
For example, the following command enters the accuracy score into the context for that experiment. This value is visible in Kubeflow and MLRun.
context.log_result(‘accuracy’,score)
deploy-inference-function.ipynb
The last step in the pipeline is to deploy the model as a serverless function for continuous inferencing. This notebook invokes the creation of the serverless function defined in nuclio-inference- function.ipynb.
Review and Build Pipeline
The combination of running all the notebooks in a pipeline enables the continuous run of experiments to reassess the accuracy of the model against new metrics. First, open the pipeline.ipynb notebook. We take you through details that show how NetApp and Iguazio simplify the deployment of this ML pipeline.
We use MLRun to provide context and handle resource allocation to each step of the pipeline. The MLRun API service runs in the Iguazio platform and is the point of interaction with Kubernetes resources. Each developer cannot directly request resources; the API handles the requests and enables access controls.
# MLRun API connection definition mlconf.dbpath = 'http://mlrun-api:8080'
The pipeline can work with NetApp Cloud Volumes and on-premises volumes. We built this demonstration to use Cloud Volumes, but you can see in the code the option to run on-premises.
# Initialize the NetApp snap fucntion once for all functions in a notebook
if [ NETAPP_CLOUD_VOLUME ]:
snapfn = code_to_function('snap',project='NetApp',kind='job',filename="snap_cv.ipynb").apply(mount_v3io())
snap_params = {
"metrics_table" : metrics_table,
"NETAPP_MOUNT_PATH" : NETAPP_MOUNT_PATH,
'MANAGER' : MANAGER,
'svm' : svm,
'email': email,
'password': password ,
'weid': weid,
'volume': volume,
"APP_DIR" : APP_DIR
}
else:
snapfn = code_to_function('snap',project='NetApp',kind='job',filename="snapshot.ipynb").apply(mount_v3io())
….
snapfn.spec.image = docker_registry + '/netapp/pipeline:latest'
snapfn.spec.volume_mounts = [snapfn.spec.volume_mounts[0],netapp_volume_mounts]
snapfn.spec.volumes = [ snapfn.spec.volumes[0],netapp_volumes]
The first action needed to turn a Jupyter notebook into a Kubeflow step is to turn the code into a function. A function has all the specifications required to run that notebook. As you scroll down the notebook, you can see that we define a function for every step in the pipeline.
| Part of the Notebook | Description |
|---|---|
<code_to_function> |
Name of the function: |
image |
The name of the Docker image we are using for this step. We created this earlier with the create-image.ipynb notebook. |
volume_mounts & volumes |
Details to mount the NetApp Cloud Volume at run time. |
We also define parameters for the steps.
params={ "FEATURES_TABLE":FEATURES_TABLE,
"SAVE_TO" : SAVE_TO,
"metrics_table" : metrics_table,
'FROM_TSDB': 0,
'PREDICTIONS_TABLE': PREDICTIONS_TABLE,
'TRAIN_ON_LAST': '1d',
'TRAIN_SIZE':0.7,
'NUMBER_OF_SHARDS' : 4,
'MODEL_FILENAME' : 'netops.v3.model.pickle',
'APP_DIR' : APP_DIR,
'FUNCTION_NAME' : 'netops-inference',
'PROJECT_NAME' : 'netops',
'NETAPP_SIM' : NETAPP_SIM,
'NETAPP_MOUNT_PATH': NETAPP_MOUNT_PATH,
'NETAPP_PVC_CLAIM' : NETAPP_PVC_CLAIM,
'IGZ_CONTAINER_PATH' : IGZ_CONTAINER_PATH,
'IGZ_MOUNT_PATH' : IGZ_MOUNT_PATH
}
After you have the function definition for all steps, you can construct the pipeline. We use the kfp module to make this definition. The difference between using MLRun and building on your own is the simplification and shortening of the coding.
The functions we defined are turned into step components using the as_step function of MLRun.
Snapshot Step Definition
Initiate a Snapshot function, output, and mount v3io as source:
snap = snapfn.as_step(NewTask(handler='handler',params=snap_params), name='NetApp_Cloud_Volume_Snapshot',outputs=['snapVolumeDetails','training_parquet_file']).apply(mount_v3io())
| Parameters | Details |
|---|---|
NewTask |
NewTask is the definition of the function run. |
(MLRun module) |
Handler. Name of the Python function to invoke. We used the name handler in the notebook, but it is not required. |
as_step |
Name. Name of the Kubeflow pipeline step. |
prep = data_prep.as_step(name='data-prep', handler='handler',params=params,
inputs = {'DATA_DIR': snap.outputs['snapVolumeDetails']} ,
out_path=artifacts_path).apply(mount_v3io()).after(snap)
| Parameters | Details |
|---|---|
inputs |
You can pass to a step the outputs of a previous step. In this case, snap.outputs['snapVolumeDetails'] is the name of the Snapshot copy we created on the snap step. |
out_path |
A location to place artifacts generating using the MLRun module log_artifacts. |
You can run pipeline.ipynb from top to bottom. You can then go to the Pipelines tab from the Iguazio dashboard to monitor progress as seen in the Iguazio dashboard Pipelines tab.
Because we logged the accuracy of training step in every run, we have a record of accuracy for each experiment, as seen in the record of training accuracy.
If you select the Snapshot step, you can see the name of the Snapshot copy that was used to run this experiment.
The described step has visual artifacts to explore the metrics we used. You can expand to view the full plot as seen in the following image.
The MLRun API database also tracks inputs, outputs, and artifacts for each run organized by project. An example of inputs, outputs, and artifacts for each run can be seen in the following image.
For each job, we store additional details.
There is more information about MLRun than we can cover in this document. Al artifacts, including the definition of the steps and functions, can be saved to the API database, versioned, and invoked individually or as a full project. Projects can also be saved and pushed to Git for later use. We encourage you to learn more at the MLRun GitHub site.