NVIDIA Base Command Manager REST data collector
The NVIDIA Base Command Manager (BCM) data collector gathers inventory, host performance, and GPU workload metrics from a BCM head node through its HTTPS REST and JSON APIs (CMDaemon, default port 8081). HYBRID mode can also collect DCGM exporter GPU metrics through /exporter. Configure one data source per cluster. This collector is in Preview; use a dedicated BCM service account rather than root or the BCM admin profile.
Terminology
For each asset type acquired by Data Infrastructure Insights, the most common terminology used for the asset is shown. When viewing or troubleshooting this data collector, keep the following terminology in mind:
| Vendor/Model Term | Data Infrastructure Insights Term |
|---|---|
BCM cluster |
Vendor Specific Metrics — |
Compute / head node |
Host (inventory) |
Slurm job or chargeback row |
Vendor Specific Metrics — |
GPU metrics (BCM GPU sampler and profiling, nvidia-smi fallback, optional DCGM exporter) |
Vendor Specific Metrics — |
Node status |
Vendor Specific Metrics — |
|
|
These are common terminology mappings only and might not represent every case for this data collector. |
Configuration
| Field | Description |
|---|---|
BCM Head Node IP/Hostname |
IP address or FQDN of the BCM head node |
HTTPS Port |
CMDaemon port. The default is 8081 |
User Name |
BCM service account for Basic authentication |
Password |
Password for the service account |
Before you save the data source, confirm the acquisition unit can reach https://<head>:8081 and that these endpoints return HTTP 200 with data:
curl -sk -u user:pass https://<head>:8081/rest/v1/device
curl -sk -u user:pass "https://<head>:8081/rest/v1/monitoring/latest?measurable=CPUUsage"
Vendor Specific Metrics use the host inventory from the latest inventory poll, so a newly added node appears there after the next inventory poll (up to 60 minutes by default).
Advanced configuration
The UI Advanced screen lists settings from all collector packages. Poll intervals are separate mandatory fields per package (not hidden); you will typically see all three below together with optional expert properties.
| Field | Description |
|---|---|
Inventory Poll Interval (min) |
Foundation / inventory package. The default is 60 minutes |
Performance Poll Interval (sec) |
Host performance package. The default is 300 seconds |
Integration Poll Interval (sec) |
Vendor Specific Metrics package. The default is 300 seconds |
Collection Mode |
REST_ONLY (default) uses only the REST and JSON APIs. HYBRID also reads |
Exporter Metric Allowlist |
HYBRID only. Optional comma-separated |
Client Certificate Path |
HYBRID only. Path on the acquisition unit to the BCM client certificate ( |
Client Key Path |
HYBRID only. Path on the acquisition unit to the certificate's key ( |
NVIDIA SMI REST integration |
Fallback GPU source. Calls |
Service account (BCM head node)
As root on the head node, create a read-only user for collection with the built-in readonly profile:
cmsh -c "user; add dii-monitor; set password '<password>'; set profile readonly; commit"
Use dii-monitor and its password in the data source. The built-in readonly profile covers inventory, the JSON API, /rest/v1/monitoring/latest (including the GPU sampler metrics), and GPU profiling data. No token is needed for this default setup. Slurm chargeback and the nvidia-smi fallback need extra tokens; see Chargeback token (optional) and nvidia-smi token (optional).
No other head node change is needed for the default REST_ONLY mode.
Chargeback token (optional)
Skip this section if you do not need Slurm chargeback data. Without the token, /chargeback returns 401. The collector logs a warning and the rest of the poll continues.
Chargeback (/chargeback, requires CHARGE_BACK_TOKEN) adds Slurm job and chargeback rows to nvidia_bcm.workload. As root on the head node, clone readonly into a profile that holds the token, and move the service account to that profile:
cmsh -c "profile; clone readonly dii-readonly; append tokens CHARGE_BACK_TOKEN; commit"
cmsh -c "user; use dii-monitor; set profile dii-readonly; commit"
Keep the data source user name and password unchanged. The stock readonly profile is not changed. The nvidia-smi token and the HYBRID steps below add their tokens to this same dii-readonly profile.
nvidia-smi token (optional)
Skip this section unless you want the nvidia-smi fallback. /nvidia-smi (requires RUN_NVIDIA_SMI_TOKEN) is a fallback GPU source. The collector calls it only when /monitoring/latest returns no GPU sampler rows. Without the token, it returns 401, the collector logs a warning, and the rest of the poll continues.
As root on the head node, add the token to the dii-readonly profile from the previous section. If you skipped that section, run its clone command first:
cmsh -c "profile; use dii-readonly; append tokens RUN_NVIDIA_SMI_TOKEN; commit"
Enabling GPU profiling metrics (optional)
BCM collects GPU profiling metrics (such as SM and tensor activity) only after you turn them on. Once they are on, the collector reports them on nvidia_bcm.gpu as gpu_profiling_*. As root on the head node:
cmsh -c "device; gpuprofiling watch -c <gpu-category> 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010"
No restart is needed, and BCM starts collecting within a few minutes. If the command reports not initialized yet, run it again a minute later. The setting does not survive a GPU node reboot; run the command again after the nodes restart. To turn profiling off, run the same command without the numbers.
Enabling /exporter for HYBRID mode
Only needed if Collection Mode is HYBRID. Leave PrometheusExporterRequireCertificate at its default; the collector logs in to /exporter with a BCM client certificate. As root on the head node:
-
Make sure dcgm-exporter runs on each GPU node (port 9400), and register it so its data reaches
/exporter:cmsh -c "device; use <node>; prometheusmetricforwarders; add dcgm-<node>; set urls http://127.0.0.1:9400/metrics; commit" -
In
/cm/local/apps/cmd/etc/cmd.conf, setEnablePrometheusExporterService = true, then runsystemctl restart cmd. -
Give the service profile the exporter token, and create a certificate for it. The service profile is
dii-readonlyfrom Chargeback token (optional); create it there first if you skipped that section:cmsh -c "profile; use dii-readonly; append tokens PROMETHEUS_EXPORTER_TOKEN; commit" cmsh -c "cert; createcertificate 2048 dii-monitor <org> <unit> <city> <state> <country> dii-readonly dii-monitor 365 /root/dii-monitor.key /root/dii-monitor.pem" -
Copy
dii-monitor.pemanddii-monitor.keyto the acquisition unit, and enter their paths in Client Certificate Path and Client Key Path.
To check from the acquisition unit (a number above 0 means it works):
curl -sk --cert dii-monitor.pem --key dii-monitor.key https://<head>:8081/exporter | grep -ci '^dcgm_'
The certificate is valid for 365 days; create a new one before it expires.
Troubleshooting
| Problem | Try this |
|---|---|
HTTP 401 |
Check the user name and password. If only |
Warning |
Set Client Certificate Path and Client Key Path, and check that the certificate's profile has |
Warning |
Fix the file named in the warning. Both files must be readable on the acquisition unit, the key must belong to the certificate, and the certificate must not be expired. |
Warning |
The GPU sampler returned no rows, so the collector used the nvidia-smi fallback, and that failed. For a 401 or 403, the service profile has no |
Warning |
Set |
No performance data |
Check that |
No GPU metrics |
Check that |
No |
Run the |
No workload / chargeback in UI |
Confirm Slurm jobs have run and chargeback is enabled; idle clusters often have empty APIs. Ensure |
Additional information on this data collector may be found from the Support page or in the Data Collector Support Matrix.