Configuring an Agent on Kubernetes

Contributors netapp-alavoie

Cloud Insights uses Telegraf as its agent for collection of integration data. Telegraf is a plugin-driven server agent that can be used to collect and report metrics, events, and logs. Input plugins are used to collect the desired information into the agent by accessing the system/OS directly, by calling third-party APIs, or by listening to configured streams (i.e. Kafka, statsD, etc). Output plugins are used to send the collected metrics, events, and logs from the agent to Cloud Insights.

Note For accurate audit and data reporting, it is strongly recommended to synchronize the time on the Agent machine using Network Time Protocol (NTP) or Simple Network Time Protocol (SNTP).
Note If you want to verify the installation files before installing the Agent, read about Verifying Kubernetes Checksums.

Operator-based install or script-based install?

Cloud Insights has built a Kubernetes Operator for monitoring customers' Kubernetes clusters. The NetApp Kubernetes Monitoring Operator (NKMO) is an improvement over script-installed monitoring methods because it allows more flexible configuration of monitoring controllable from Cloud Insights and fewer customer-driven configuration interventions, as well as enhances opportunities for monitoring other software running in the K8s cluster.

NKMO continues to use underlying telegraf software for data collection, transformation, and delivery to Cloud Insights. This is enhanced with Custom Resource Definitions and Custom Resources to tailor monitoring for each K8s cluster.

Installing the agent

If you are installing a Service data collector and have not yet configured an Agent, you are prompted to first install an Agent for the appropriate Operating System. This topic provides instructions for installing the Telegraf agent on the following Operating Systems:

To install an agent, you must first do the following:

  1. Log into the host you will use for your agent.

  2. Log in to your Cloud Insights site and go to Admin > Data Collectors.

  3. Click on +Data Collector and choose a data collector to install.

  4. Choose the Kubernetes platform for your host

  5. Follow the remaining steps.

To install an agent on Windows, Linux, or Mac, follow these instructions.

Note Once you have installed an agent on a host, you do not need to install an agent again on that host.
Tip Once you have installed an agent on a server/VM, Cloud Insights collects metrics from that system in addition to collecting from any data collectors you configure. These metrics are gathered as "Node" metrics.
Note If you are using a proxy, read the proxy instructions for your platform before installing the Telegraf agent.

Installing an Agent on Kubernetes

Kubernetes offers two ways to collect data:

  • NetApp Kubernetes Monitoring Operator configuration. This is the recommended installation method for Kubernetes.

  • Traditional script-based Agent installation (not recommended)

Installation instructions vary based on which you choose.

Kubernetes Installation Choices

Pre-requisites:
  • NetApp Kubernetes Monitoring Operator installation is supported with Kubernetes 1.17 or greater. When used with the Docker container runtime, Cloud Insights can display pod-to-PV-to-storage mappings for NFS and iSCSI; other runtimes only show iSCSI.

  • If you are running on OpenShift 4.6 - 4.8, you must follow the OpenShift Instructions below in addition to ensuring these pre-requisites are met.

  • The following commands must be available: curl, sudo (not required for script-based installation), openssl, sha256sum, and kubectl. For best results, add these commands to the PATH.

  • kube-state-metrics must be installed. See below for more information. kube-state-metrics is automatically installed with Operator-based installation.

  • If you are behind a proxy, follow the instructions in the Configuring Proxy Support for Kubernetes section.

  • If you are running a Kubernetes variant that requires security context constraints, follow the instructions in the Configuring the Agent to Collect Data from Kubernetes section. Operator-based installation installs this for you.

  • You must have permissions to create Kubernetes cluster roles and role bindings.

Monitoring is only installed on Linux nodes

Cloud Insights supports monitoring of Kubernetes nodes that are running Linux, by specifying a Kubernetes node selector that looks for the following Kubernetes labels on these platforms:

Platform Label

Kubernetes v1.17 and above

Kubernetes.io/os = linux

Rancher + cattle.io as orchestration/Kubernetes platform

cattle.io/os = linux

NetApp Kubernetes Monitoring Operator Installation

Operator-Based Install

Steps to install NetApp Kubernetes Monitoring Operator agent on Kubernetes:
  1. Enter the cluster name and namespace.

  2. Once these are entered, you can copy the Agent Installer snippet

  3. Click the button to copy this snippet to the clipboard.

  4. Paste the snippet into a bash window and execute it.

  5. The installation proceeds automatically. When it is complete, click the Complete Setup button.

OpenShift Instructions

If you are running on OpenShift 4.6 - 4.8, you must change the "privileged-mode" setting. Run the following command to open the agent for editing. If you are using a namespace other than "netapp-monitoring", specify that namespace in the command line:

kubectl edit agent agent-monitoring-netapp -n netapp-monitoring

In the file, change privileged-mode: false to privileged-mode: true

Configuring Proxy Support for NetApp Kubernetes Monitoring Operator

To configure a proxy for the monitoring operator, perform the following steps.

First, open the agent-monitoring-netapp file for editing:

kubectl -n netapp-monitoring edit agent agent-monitoring-netapp

In the spec: section of this file, add the following code block:

spec:
  proxy:
    isAuProxyEnabled: <true or false>
    isTelegrafProxyEnabled: <true or false>
    isFluentbitProxyEnabled: <true or false>
    password: <password for proxy, optional>
    port: <port for proxy>
    server: <server for proxy>
    username: <username for proxy, optional>
    noProxy: <comma separated list of IPs or resolvable hostnames that should bypass a proxy>

Using a custom/private docker repository

If using a custom docker repository, do the following:

Get the docker secret:

kubectl -n netapp-monitoring get secret docker -o yaml

Copy/paste the value of .dockerconfigjson: from the output of the above command.

Decode the docker secret:

echo <paste from _.dockerconfigjson:_  output above> | base64 -d

The output of this will be in the following json format:

{ "auths":
  {"docker.<cluster>.cloudinsights.netapp.com" :
    {"username":"<tenant id>",
     "password":"<password which is the CI API key>",
     "auth"    :"<encoded username:password basic auth key. This is internal to docker>"}
  }
}

Log in to the docker repository:

docker login docker.<cluster>.cloudinsights.netapp.com (from step #2) -u <username from step #2>
password: <password from docker secret step above>

Pull the operator docker image from Cloud Insights:

docker pull docker.<cluster>.cloudinsights.netapp.com/netapp-monitoring:<version>

Find the <version> field using the following command:

kubectl -n netapp-monitoring get deployment monitoring-operator | grep "image:"

Push the operator docker image to your private/local/enterprise docker repository according to your corporate policies.

Download all open source dependencies to your private docker registry. The following open source images need to be downloaded:

docker.io/telegraf:1.21.4
gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0

If fluent-bit is enabled, also download:

docker.io/fluent-bit:1.8.12
docker.io/kubernetes-event-exporter:0.10

Edit the agent CR to reflect the new docker repo location, disable auto upgrade (if enabled).

kubectl -n netapp-monitoring edit agent agent-monitoring-netapp
enableAutoUpgrade: false
docker-repo: <docker repo of the enterprise/corp docker repo>
dockerRepoSecret: <optional: name of the docker secret of enterprise/corp docker repo, this secret should be already created on the k8s cluster in the same namespace>

In the spec: section, make the following changes:

spec:
  telegraf:
    - name: ksm
      substitutions:
        - key: k8s.gcr.io
          value: <same as "docker-repo" field above>

Upgrading from Script-based K8s monitoring to Operator-based

If you already have installed script-based Kubernetes monitoring, follow these steps to upgrade to operator-based monitoring:

Steps to upgrade

  1. Preserve the ConfigMap from the script-based monitoring installation:

    kubectl --namespace ci-monitoring get cm -o yaml > /tmp/telegraf-configs.yaml
  2. Save the K8s cluster name for use during installation of the K8s operator-based monitoring solution to ensure data continuity.

    If you do not remember the name of the K8s cluster in CI, it can be extracted from your saved configuration with the following command line:

    cat /tmp/telegraf-configs.yaml | grep kubernetes_cluster | head -2
  3. Remove the script-based monitoring

    To uninstall the script-based agent on Kubernetes, do the following:

    If the monitoring namespace is being used solely for Telegraf:

    kubectl --namespace ci-monitoring delete ds,rs,cm,sa,clusterrole,clusterrolebinding -l app=ci-telegraf
    kubectl delete ns ci-monitoring

    If the monitoring namespace is being used for other purposes in addition to Telegraf:

    kubectl --namespace ci-monitoring delete ds,rs,cm,sa,clusterrole,clusterrolebinding -l app=ci-telegraf
Install the K8s operator-based monitoring following the instructions in the install tile.

Tile for Kubernetes Operator

Script-Based Installation

Note Script-based installation is deprecated. Please use Kubernetes Operator-based collection for monitoring your Kubernetes cluster.

Script-Based Install

Steps to install Script-based agent on Kubernetes:
  1. Choose an Agent Access Key.

  2. Click the Copy Agent Installer Snippet button in the installation dialog. You can optionally click the +Reveal Agent Installer Snippet button if you want to view the command block.

  3. Paste the command into a bash window.

  4. Optionally, you can override the namespace or provide the cluster name as part of the install command by modifying the command block to add one or both of the following before the final ./$installerName

    • CLUSTER_NAME=<Cluster Name>

    • NAMESPACE=<Namespace>

      Here it is in place in the command block:

      installerName=cloudinsights-kubernetes.sh ... && CLUSTER_NAME=<cluster_name> NAMESPACE=<new_namespace> sudo -E -H ./$installerName --download --install
      Tip CLUSTER_NAME is the name of the Kubernetes cluster from Cloud Insights collects metrics, while NAMESPACE is the namespace to which the Telegraf agent will be deployed. The specified namespace will be created if it does not already exist.
  5. When ready, execute the command block.

  6. The command will download the appropriate agent installer, install it, and set a default configuration. If you have not explicitly set the namespace, you will be prompted to enter it. When finished, the script will restart the agent service. The command has a unique key and is valid for 24 hours.

  7. When finished, click Complete Setup.

Configuring Proxy Support for Kubernetes - Script-Based

Note The steps below outline the actions needed to set the http_proxy/https_proxy environment variables. For some proxy environments, users may also need to set the no_proxy environment variable.

For systems residing behind a proxy, perform the following to set the https_proxy and/or http_proxy environment variable(s) for the current user PRIOR to installing the Telegraf agent:

export https_proxy=<proxy_server>:<proxy_port>

AFTER installing the Telegraf agent, add and set the appropriate https_proxy and/or http_proxy environment variable(s) to the telegraf-ds daemonset and telegraf-rs replicaset.

kubectl edit ds telegraf-ds
…
       env:
       - name: https_proxy
         value: <proxy_server>:<proxy_port>
       - name: HOSTIP
         valueFrom:
           fieldRef:
             apiVersion: v1
             fieldPath: status.hostIP
…
kubectl edit rs telegraf-rs
…
       env:
       - name: https_proxy
         value: <proxy_server>:<proxy_port>
       - name: HOSTIP
         valueFrom:
           fieldRef:
             apiVersion: v1
             fieldPath: status.hostIP
…

Then, restart Telegraf:

kubectl delete pod telegraf-ds-*
kubectl delete pod telegraf-rs-*

DaemonSet, ReplicaSet, and Stopping/Starting the agent

A DaemonSet and ReplicaSet will be created on the Kubernetes cluster to run the required Telegraf agents/pods. By default, these Telegraf agents/pods will be scheduled on both master and non-master nodes.

To facilitate stopping and restarting of the agent, generate the Telegraf DaemonSet YAML and ReplicaSet YAML using the following commands. Note that these commands are using the default namespace "ci-monitoring". If you have set your own namespace, substitute that namespace in these and all subsequent commands and files:

If you have set your own namespace, substitute that namespace in these and all subsequent commands and files:

kubectl --namespace ci-monitoring get ds telegraf-ds -o yaml > /tmp/telegraf-ds.yaml
kubectl --namespace ci-monitoring get rs telegraf-rs -o yaml > /tmp/telegraf-rs.yaml

You can then use the following commands to stop and start the Telegraf service:

kubectl --namespace ci-monitoring delete ds telegraf-ds
kubectl --namespace ci-monitoring delete rs telegraf-rs
kubectl --namespace ci-monitoring apply -f /tmp/telegraf-ds.yaml
kubectl --namespace ci-monitoring apply -f /tmp/telegraf-rs.yaml

Configuring the Agent to Collect Data from Kubernetes

Note: The default namespace for Script-based installation is ci-monitoring. For Operator-based installation, the default namespace is netapp-monitoring. In commands involving namespace, be sure to specify the correct namespace for your installation.

The pods in which the agents run need to have access to the following:

  • hostPath

  • configMap

  • secrets

These Kubernetes objects are automatically created as part of the Kubernetes agent install command provided in the Cloud Insights UI. Some variants of Kubernetes, such as OpenShift, implement an added level of security that may block access to these components. The SecurityContextConstraint is not created as part of the Kubernetes agent install command provided in the Cloud Insights UI, and must be created manually. Once created, restart the Telegraf pod(s).

    apiVersion: v1
    kind: SecurityContextConstraints
    metadata:
      name: telegraf-hostaccess
      creationTimestamp:
      annotations:
        kubernetes.io/description: telegraf-hostaccess allows hostpath volume mounts for restricted SAs.
      labels:
        app: ci-telegraf
    priority: 10
    allowPrivilegedContainer: true
    defaultAddCapabilities: []
    requiredDropCapabilities: []
    allowedCapabilities: []
    allowedFlexVolumes: []
    allowHostDirVolumePlugin: true
    volumes:
    - hostPath
    - configMap
    - secret
    allowHostNetwork: false
    allowHostPorts: false
    allowHostPID: false
    allowHostIPC: false
    seLinuxContext:
      type: MustRunAs
    runAsUser:
      type: RunAsAny
    supplementalGroups:
      type: RunAsAny
    fsGroup:
      type: RunAsAny
    readOnlyRootFilesystem: false
    users:
    - system:serviceaccount:ci-monitoring:monitoring-operator
    groups: []

Installing the kube-state-metrics server

Note Operator-based install handles the installation of kube-state-metrics. Skip this section if you are performing Operator-based installation.
Note It is strongly recommended to use kube-state-metrics version 2.0 or later in order to take advantage of the full feature set including the ability to link Kubernetes persistent volumes (PVs) to backend storage devices. Note also that with kube-state-metrics version 2.0 and above, Kubernetes object labels are not exported by default. To configure kube-state-metrics to export Kubernetes object labels, you must specify a metric labels "allow" list. Refer to the --metric-labels-allowlist option in the kube-state-metrics documentation.

Use the following steps to install the kube-state-metrics server (required if you are performing script-based installation):

Steps
  1. Create a temporary folder (for example, /tmp/kube-state-yaml-files/) and copy the .yaml files from https://github.com/kubernetes/kube-state-metrics/tree/master/examples/standard to this folder.

  2. Run the following command to apply the .yaml files needed for installing kube-state-metrics:

    kubectl apply -f /tmp/kube-state-yaml-files/

Uninstalling the Agent

Note that these commands are using the default namespace "ci-monitoring". If you have set your own namespace, substitute that namespace in these and all subsequent commands and files.

To uninstall the script-based agent on Kubernetes, do the following:

If the monitoring namespace is being used solely for Telegraf:

kubectl --namespace ci-monitoring delete ds,rs,cm,sa,clusterrole,clusterrolebinding -l app=ci-telegraf
kubectl delete ns ci-monitoring

If the monitoring namespace is being used for other purposes in addition to Telegraf:

kubectl --namespace ci-monitoring delete ds,rs,cm,sa,clusterrole,clusterrolebinding -l app=ci-telegraf

For Operator-based installation run the following commands:

kubectl delete ns netapp-monitoring
kubectl delete agent agent-monitoring-netapp
kubectl delete crd agents.monitoring.netapp.com
kubectl delete role agent-leader-election-role
kubectl delete clusterrole agent-manager-role agent-proxy-role agent-metrics-reader
kubectl delete clusterrolebinding agent-manager-rolebinding agent-proxy-rolebinding agent-cluster-admin-rolebinding

If a Security Context Constraint was previously-created manually for a script-based Telegraf installation:

kubectl delete scc telegraf-hostaccess

Upgrading the Agent

Note that these commands are using the default namespace "ci-monitoring". If you have set your own namespace, substitute that namespace in these and all subsequent commands and files.

To upgrade the telegraf agent, do the following:

  1. Back up the existing configurations:

    kubectl --namespace ci-monitoring get cm -o yaml > /tmp/telegraf-configs.yaml
  1. Uninstall the Agent (see above for instructions)

  2. Install the new agent.

Verifying Kubernetes Checksums

The Cloud Insights agent installer performs integrity checks, but some users may want to perform their own verifications before installing or applying downloaded artifacts. To perform a download-only operation (as opposed to the default download-and-install), these users can edit the agent installation command obtained from the UI and remove the trailing “install” option.

Follow these steps:

  1. Copy the Agent Installer snippet as directed.

  2. Instead of pasting the snippet into a command window, paste it into a text editor.

  3. Remove the trailing “--install” (Linux/Mac) or “-install” (Windows) from the command.

  4. Copy the entire command from the text editor.

  5. Now paste it into your command window (in a working directory) and run it.

Non-Windows (these examples are for Kubernetes; actual script names may vary):

  • Download and install (default):

    installerName=cloudinsights-kubernetes.sh … && sudo -E -H ./$installerName --download –-install
  • Download-only:

    installerName=cloudinsights-kubernetes.sh … && sudo -E -H ./$installerName --download

The download-only command will download all required artifacts from Cloud Insights to the working directory. The artifacts include, but may not be limited to:

  • an installation script

  • an environment file

  • YAML files

  • a signed checksum file (sha256.signed)

  • a PEM file (netapp_cert.pem) for signature verification

The installation script, environment file, and YAML files can be verified using visual inspection.

The PEM file can be verified by confirming its fingerprint to be the following:

E5:FB:7B:68:C0:8B:1C:A9:02:70:85:84:C2:74:F8:EF:C7:BE:8A:BC

More specifically,

  • Non-Windows:

    openssl x509 -fingerprint -sha1 -noout -inform pem -in netapp_cert.pem
  • Windows:

    Import-Certificate -Filepath .\netapp_cert.pem -CertStoreLocation Cert:\CurrentUser\Root

The signed checksum file can be verified using the PEM file:

  • Non-Windows:

    openssl smime -verify -in sha256.signed -CAfile netapp_cert.pem -purpose any
  • Windows (after installing the certificate via Import-Certificate above):

    Get-AuthenticodeSignature -FilePath .\sha256.ps1 $result = Get-AuthenticodeSignature -FilePath .\sha256.ps1 $signer = $result.SignerCertificate Add-Type -Assembly System.Security [Security.Cryptography.x509Certificates.X509Certificate2UI]::DisplayCertificate($signer)

Once all of the artifacts have been satisfactorily verified, the agent installation can be initiated by running:

Non-Windows:

sudo -E -H ./<installation_script_name> --install

Windows:

.\cloudinsights-windows.ps1 -install

Troubleshooting Kubernetes Agent Installation

Some things to try if you encounter problems setting up an agent:

Problem: Try this:

For clusters where etcd is not the Kubernetes cluster datastore, You will see the following message in the telegraf RS pod:

[inputs.prometheus] Error in plugin: could not load keypair /etc/kubernetes/pki/etcd/server.crt:/etc/kubernetes/pki/etcd/server.key: open /etc/kubernetes/pki/etcd/server.crt: no such file or directory

Cloud Insights only supports monitoring of etcd as the K8s datastore. You can modify the agent to avoid collecting etcd data by changing the configuration with the following instructions:

kubectl -n netapp-monitoring edit agent agent-monitoring-netapp

In that file, delete the following section:

- name: prometheus_etcd
run-mode:
- ReplicaSet

I already installed an agent using Cloud Insights

If you have already installed an agent on your host/VM, you do not need to install the agent again. In this case, simply choose the appropriate Platform and Key in the Agent Installation screen, and click on Continue or Finish.

I already have an agent installed but not by using the Cloud Insights installer

Remove the previous agent and run the Cloud Insights Agent installation, to ensure proper default configuration file settings. When complete, click on Continue or Finish.

I do not see a hyperlink/connection between my Kubernetes Persistent Volume and the corresponding back-end storage device. My Kubernetes Persistent Volume is configured using the hostname of the storage server.

Follow the steps to uninstall the existing Telegraf agent, then re-install the latest Telegraf agent. You must be using Telegraf version 2.0 or later.

I’m seeing messages in the logs resembling the following:

E0901 15:21:39.962145 1 reflector.go:178] k8s.io/kube-state-metrics/internal/store/builder.go:352: Failed to list *v1.MutatingWebhookConfiguration: the server could not find the requested resource
E0901 15:21:43.168161 1 reflector.go:178] k8s.io/kube-state-metrics/internal/store/builder.go:352: Failed to list *v1.Lease: the server could not find the requested resource (get leases.coordination.k8s.io)
etc.

These messages may occur if you are running kube-state-metrics version 2.0.0 or above with Kubernetes version 1.17 or below.


To get the Kubernetes version:

kubectl version

To get the kube-state-metrics version:

kubectl get deploy/kube-state-metrics -o jsonpath='{..image}'

To prevent these messages from happening, users can modify their kube-state-metrics deployment to disable the following Leases:

mutatingwebhookconfigurations
validatingwebhookconfigurations
volumeattachments resources

More specifically, they can use the following CLI argument:

resources=certificatesigningrequests,configmaps,cronjobs,daemonsets, deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges, namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes, poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas, secrets,services,statefulsets,storageclasses

The default resource list is:

"certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments, endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges, mutatingwebhookconfigurations,namespaces,networkpolicies,nodes, persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets, replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses, validatingwebhookconfigurations,volumeattachments"

I installed or upgraded Telegraf on Kubernetes, but the Telegraf pods are not starting up. The Telegraf ReplicaSet or DaemonSet is reporting a failure resembling the following:

Error creating: pods "telegraf-rs-" is forbidden": unable to validate against any security context constraint: [spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used]

Create a Security Context Constraint (refer to the Configuring the Agent to Collect Data from Kubernetes section above) if one does not already exist.

Ensure the namespace and service account specified for the Security Context Constraint matches the namespace and service account for the Telegraf ReplicaSet and DaemonSet.

kubectl describe scc telegraf-hostaccess |grep serviceaccount
kubectl -n ci-monitoring --describe rs telegraf-rs | grep -i "Namespace:"
kubectl -n ci-monitoring describe rs telegraf-rs | grep -i "Service Account:"
kubectl -n ci-monitoring --describe ds telegraf-ds | grep -i "Namespace:"
kubectl -n ci-monitoring describe ds telegraf-ds | grep -i "Service Account:"

I see error messages from Telegraf resembling the following, but Telegraf does start up and run:

Oct 11 14:23:41 ip-172-31-39-47 systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB.
Oct 11 14:23:41 ip-172-31-39-47 telegraf[1827]: time="2021-10-11T14:23:41Z" level=error msg="failed to create cache directory. /etc/telegraf/.cache/snowflake, err: mkdir /etc/telegraf/.ca
che: permission denied. ignored\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"
Oct 11 14:23:41 ip-172-31-39-47 telegraf[1827]: time="2021-10-11T14:23:41Z" level=error msg="failed to open. Ignored. open /etc/telegraf/.cache/snowflake/ocsp_response_cache.json: no such
file or directory\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"
Oct 11 14:23:41 ip-172-31-39-47 telegraf[1827]: 2021-10-11T14:23:41Z I! Starting Telegraf 1.19.3

This is a known issue. Refer to This GitHub article for more details. As long as Telegraf is up and running, users can ignore these error messages.

On Kubernetes, my Telegraf pod(s) are reporting the following error:
"Error in processing mountstats info: failed to open mountstats file: /hostfs/proc/1/mountstats, error: open /hostfs/proc/1/mountstats: permission denied"

If SELinux is enabled and enforcing, it is likely preventing the Telegraf pod(s) from accessing the /proc/1/mountstats file on the Kubernetes nodes. To relax this restriction, do ONE of the following:

• For script-based installations, edit the telegraf DS (kubectl edit ds telegraf-ds), and change "privileged: false" to "privileged: true"
• For operator-based installation, edit the agent (kubectl edit agent agent-monitoring-netapp), and change "privileged-mode: false" to "privileged-mode: true"

On Kubernetes, my Telegraf ReplicaSet pod is reporting the following error:

[inputs.prometheus] Error in plugin: could not load keypair /etc/kubernetes/pki/etcd/server.crt:/etc/kubernetes/pki/etcd/server.key: open /etc/kubernetes/pki/etcd/server.crt: no such file or directory

The Telegraf ReplicaSet pod is intended to run on a node designated as a master or for etcd. If the ReplicaSet pod is not running on one of these nodes, you will get these errors. Check to see if your master/etcd nodes have taints on them. If they do, add the necessary tolerations to the Telegraf ReplicaSet, telegraf-rs.

For example, edit the ReplicaSet…​

kubectl edit rs telegraf-rs

…​and add the appropriate tolerations to the spec. Then, restart the ReplicaSet pod.

Additional information may be found from the Support page or in the Data Collector Support Matrix.