On the admin host, install the Kerberos client tools and cifs-utils.

On every other node, install cifs-utils.

On every node, verify the installation.

which cifs.upcall

Set the namespace that AI Data Engine runs in, and verify that the configuration volume claim is bound. The default namespace is aide.

export NAMESPACE="<namespace>"
kubectl -n "${NAMESPACE}" get pvc aide-config-pvc

Get the NFS server and the export path of the configuration volume. You specified these values when you deployed AI Data Engine.

kubectl get pv aide-config-pv -o jsonpath='NFS server: {.spec.nfs.server}{"\n"}Export path: {.spec.nfs.path}{"\n"}'

Mount the export. Replace <nfs_server> and <export_path> with the values from the previous step. The commands use /mnt/aide-config as the mount point. You can use any empty directory.

mkdir -p /mnt/aide-config
mount -t nfs <nfs_server>:<export_path> /mnt/aide-config

Create the Kerberos directories.

export AIDE_CONFIG_ROOT="/mnt/aide-config"
mkdir -p "${AIDE_CONFIG_ROOT}/kerberos/kerberos_refresher"
mkdir -p "${AIDE_CONFIG_ROOT}/kerberos/kerberos_ccache"

Set the variables that the following commands use. Replace each value in angle brackets with your value, and don't include the angle brackets. Enter the realm in uppercase and the domain in lowercase.

export KDC_IP="<kdc_ip>"
export KDC_FQDN="<kdc_fqdn>"
export AD_DOMAIN="<domain>"
export REALM="<REALM>"
export SCAN_USER="<scan_user>"
export FILE_SERVER_FQDN="<file_server_fqdn>"
export SHARE_NAME="<share_name>"
export KRB5_CONF_SOURCE="${AIDE_CONFIG_ROOT}/kerberos/krb5.conf"
export KRB5_KEYTAB="${AIDE_CONFIG_ROOT}/kerberos/kerberos_refresher/datasense.keytab"
export KRB5_CREDS="${AIDE_CONFIG_ROOT}/kerberos/kerberos_refresher/kerberos_creds.env"

Create the Kerberos configuration file on the NFS export. AI Data Engine reads this file from the export. Don't create it in /etc/krb5.conf on the nodes.

tee "${KRB5_CONF_SOURCE}" >/dev/null <<EOF
[libdefaults]
    default_realm = ${REALM}
    dns_lookup_realm = false
    dns_lookup_kdc = false
    rdns = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true

[realms]
    ${REALM} = {
        kdc = ${KDC_IP}:88
        admin_server = ${KDC_IP}:749
    }

[domain_realm]
    .${AD_DOMAIN} = ${REALM}
    ${AD_DOMAIN} = ${REALM}
EOF
chmod 644 "${KRB5_CONF_SOURCE}"

Create a keytab for the scan service account.

Test the keytab. The KRB5_CONFIG variable points the Kerberos tools at the file on the export.

export KRB5_CONFIG="${KRB5_CONF_SOURCE}"
klist -kt /tmp/krb5.keytab
kinit -kt /tmp/krb5.keytab "${SCAN_USER}@${REALM}"
klist
kdestroy

Install the keytab, and then delete the temporary copy.

install -m 640 -g 4321 /tmp/krb5.keytab "${KRB5_KEYTAB}"
rm -f /tmp/krb5.keytab

Make the ticket cache directory writable by the ticket refresher.

chown root:4321 "${AIDE_CONFIG_ROOT}/kerberos/kerberos_ccache"
chmod 770 "${AIDE_CONFIG_ROOT}/kerberos/kerberos_ccache"

Create the principal file. The KRB5_RENEW_INTERVAL_H value is the ticket renewal interval in hours.

tee "${KRB5_CREDS}" >/dev/null <<EOF
KRB5_USER=${SCAN_USER}
KRB5_REALM=${REALM}
KRB5_RENEW_INTERVAL_H=3
EOF
chmod 640 "${KRB5_CREDS}"
chgrp 4321 "${KRB5_CREDS}"

Verify that the files are in place.

ls -l "${KRB5_CONF_SOURCE}" "${KRB5_KEYTAB}" "${KRB5_CREDS}"

On every node, verify that the file server and the KDC resolve by FQDN. Replace the values with your file server FQDN and KDC FQDN.

nslookup <file_server_fqdn>
nslookup <kdc_fqdn>

Check the result.

On the admin host, start a refresh job, wait for it to complete, and display its log.

JOB="kerberos-refresh-$(date +%s)"
kubectl -n "${NAMESPACE}" create job "${JOB}" --from=cronjob/ds-krb5-ticket-refresher
kubectl -n "${NAMESPACE}" wait --for=condition=complete "job/${JOB}" --timeout=120s
kubectl -n "${NAMESPACE}" logs "job/${JOB}"

Verify that the shared ticket cache exists and has a recent timestamp.

ls -l "${AIDE_CONFIG_ROOT}/kerberos/kerberos_ccache/krb5cc_shared"

List the mount service pods and the nodes that they run on.

kubectl -n "${NAMESPACE}" get pods -l app.kubernetes.io/name=ds-mount-server -o wide

For each pod, run the following commands. Replace <pod_name> with the name of the pod.

POD=<pod_name>
kubectl -n "${NAMESPACE}" exec "${POD}" -- nslookup "${FILE_SERVER_FQDN}"
kubectl -n "${NAMESPACE}" exec "${POD}" -- klist
kubectl -n "${NAMESPACE}" exec "${POD}" -- cat /etc/krb5.conf

(Optional) Test a mount of the share by using Kerberos.

kubectl -n "${NAMESPACE}" exec "${POD}" -- sh -c "mount -t cifs -o sec=krb5,multiuser //${FILE_SERVER_FQDN}/${SHARE_NAME} /mnt && ls -l /mnt && umount /mnt"

From the AI Data Engine UI, select Configuration.

On the Scanned systems tab, select Add system.

Select File shares group.

Enter a name for the group.

Enter the share path in the format <file_server_fqdn>/<share_name>. Replace the placeholders with your values. Don't use an IP address.

For the authentication method, select Kerberos. Leave the user name and password fields empty.

Review your selections and select Confirm.

Select Activate scans for the new system.

Verify that your Active Directory DNS server also resolves external names.

On every node, back up /etc/resolv.conf, and set your Active Directory DNS server as the only name server. If you open a new shell on a node, set the KDC_IP variable again.

export KDC_IP="<kdc_ip>"
cp /etc/resolv.conf /etc/resolv.conf.bak
ORIG_SEARCH=$(grep '^search' /etc/resolv.conf.bak | head -1 | cut -d' ' -f2-)
printf 'search %s\nnameserver %s\n' "${ORIG_SEARCH}" "${KDC_IP}" > /etc/resolv.conf

On every node, prevent the network manager from overwriting the DNS settings after a restart.

From the admin host, restart CoreDNS so that AI Data Engine uses the new DNS settings. You run this step once for the cluster. CoreDNS forwards queries for names outside the cluster to the name servers in the /etc/resolv.conf file of the node, so it must restart to pick up the change.

Verify name resolution again. Refer to Verify name resolution.

On the admin host, restart the mount service, and delete the previous ticket refresh jobs.

kubectl -n "${NAMESPACE}" rollout restart daemonset ds-mount-server
kubectl -n "${NAMESPACE}" delete job -l app.kubernetes.io/name=ds-krb5-ticket-refresher --ignore-not-found

Start a new ticket refresh. Refer to Start the first ticket refresh.

From the AI Data Engine UI, delete the SMB/CIFS data source that uses Kerberos.

On the admin host, with the NFS export mounted at /mnt/aide-config, delete the Kerberos files.

rm -f /mnt/aide-config/kerberos/kerberos_refresher/datasense.keytab
rm -f /mnt/aide-config/kerberos/kerberos_refresher/kerberos_creds.env
rm -f /mnt/aide-config/kerberos/krb5.conf
rm -f /mnt/aide-config/kerberos/kerberos_ccache/krb5cc_shared
umount /mnt/aide-config

If you changed the DNS settings, restore the original settings.