A VMware VCF has been setup. For information or instruction on how to create a VCF, please refer to VMware documentation VMware Cloud Foundation Documentation.

Provision three Linux VMs, two VMs for Oracle database and one VM for Ansible controller within VCF workload domain. Provision one Windows server VM for running NetApp SnapCenter server. For information on setting up Ansible controller for automated Oracle database deployment, referring to following resources Getting Started with NetApp solution automation.

SnapCenter plugin version 6.0 for VMware vSphere has been deployed in VCF. Refer to following resources for the plugin deployment: SnapCenter Plug-in for VMware vSphere documentation.

ONTAP tool for VMware vSphere has been deployed in VCF. Refer to following resources for the ONTAP tool for VMware vSphere deployment: ONTAP tools for VMware vSphere documentation

From vSphere client shortcuts, open NetApp ONTAP tool. Ensure that ONTAP storage cluster has been added to Storage Systems as part of ONTAP tool deployment.

Click on Storage capability profile to add a custom profile for Oracle. Name the profile and add a brief description.

Choose storage controller category: performance, capacity, or hybrid.

Select the protocol.

Define a QoS policy if desired.

Additional storage attributes for the profile. Be sure that the encryption is enabled on the NetApp controller if you want to have the encryption capability or it may cause issues when applying the profile.

Review the summary and finish the storage capability profile creation.

The vVols creation workflow can be triggered either from ONTAP tool interface or from VCF workload domain cluster.

Filling in general information for datastore including provisioning destination, type, name, and protocol.

Select the custom storage capability profile created from previous step, the Storage system, and Storage VM, where vVols are to be created.

Choose Create new volumes, fill in the volume name and size and click on ADD then NEXT to move to the summary page.

You could add more than one volume to a vVols datastore or span a vVols datastore volumes across ONTAP controller nodes for performance.

Click Finish to create vVols datastore for Oracle binary.

Repeat the same procedures to create vVols datastore for Oracle data and log.

From vSphere client menus, open Policies and Profiles and highlight VM Storage Policies. Click Create to open VM Storage Policies workflow.

Name the VM storage policy.

In Datastore specific rules, check Enable rules for "NetAPP.clustered.Data.ONTAP.VP.vvol" storage

For NetApp.clustered.Data.ONTAP.VP.vvol rules Placement, select the custom storage capacity profile created from the previous step.

For NetApp.clustered.Data.ONTAP.VP.vvol rules Replication, choose Disabled if vVols are not replicated.

Storage compatibility page displays the compatible vVols datastores in VCF environment.

Review and finish to create the VM Storage Policy.

Validate the VM Storage Policy just created.

Add a disk to VM for Oracle binary storage.

Add a disk to VM for Oracle data storage.

Add a disk to VM for Oracle log storage.

From VM Edit Settings, Advanced Parameters, add Attribute disk.enableuuid with Value TRUE. The VM needs to be down to add the advanced parameter. Setting this option enables SnapCenter to precisely identify the vVol in your environment.

Now, restart the VM. Login to VM as an admin user via ssh to review the newly added disk drives.

[admin@ora_01 ~]$ sudo fdisk -l

Disk /dev/sdb: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdc: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdd: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

.
.
.

Partition the drives as a primary and single partition by simply accepting the default choices.

sudo fdisk /dev/sdb

sudo fdisk /dev/sdc

sudo fdisk /dev/sdd

Format the partitioned disks as xfs file systems.

sudo mkfs.xfs /dev/sdb1

sudo mkfs.xfs /dev/sdc1

sudo mkfs.xfs /dev/sdd1

Mount the drives to mount point /u01, /u02, and /u03.

sudo mount -t xfs /dev/sdb1 /u01

sudo mount -t xfs /dev/sdc1 /u02

sudo mount -t xfs /dev/sdd1 /u03

[admin@ora_01 ~]$ df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               7.7G     0  7.7G   0% /dev
tmpfs                  7.8G     0  7.8G   0% /dev/shm
tmpfs                  7.8G  782M  7.0G  10% /run
tmpfs                  7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/rhel-root   44G   19G   26G  43% /
/dev/sda1             1014M  258M  757M  26% /boot
tmpfs                  1.6G   12K  1.6G   1% /run/user/42
tmpfs                  1.6G  4.0K  1.6G   1% /run/user/1000
/dev/sdb1               50G  390M   50G   1% /u01
/dev/sdc1              100G  746M  100G   1% /u02
/dev/sdd1              100G  746M  100G   1% /u03

Add mount points to /etc/fstab so that disk drives will be mounted when VM reboots.

sudo vi /etc/fstab

[oracle@ora_01 ~]$ cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Wed Oct 18 19:43:31 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=aff942c4-b224-4b62-807d-6a5c22f7b623 /boot                   xfs     defaults        0 0
/dev/mapper/rhel-swap   none                    swap    defaults        0 0
/root/swapfile swap swap defaults 0 0
/dev/sdb1               /u01                    xfs     defaults        0 0
/dev/sdc1               /u02                    xfs     defaults        0 0
/dev/sdd1               /u03                    xfs     defaults        0 0

Login to Ansible controller VM as admin user via ssh and clone a copy of automation toolkit for Oracle on NFS.

git clone https://bitbucket.ngage.netapp.com/scm/ns-bb/na_oracle_deploy_nfs.git

Stage the following Oracle installation files in /tmp/archive folder on database VM. The folder should allow all user access with 777 permission.

LINUX.X64_193000_db_home.zip
p34765931_190000_Linux-x86-64.zip
p6880880_190000_Linux-x86-64.zip

Configure deployment target file - hosts, global variables file - vars/vars.yml, and local DB VM variables file - host_vars/host_name.yml according to instructions in this section of TR-4992: Parameter files configuration. Comment out nfs_lif variable from local DB VM variable file.

Set up ssh keyless authentication between Ansible controller and database VMs, which requires to generate a ssh key pair and copy the public key to database VMs admin user root directory .ssh folder authorized_keys file.

ssh-keygen

From Ansible controller, cloned automation toolkit home directory /home/admin/na_oracle_deploy_nf, execute prerequisites playbook.

ansible-playbook -i hosts 1-ansible_requirements.yml

Execute Linux configuration playbook.

ansible-playbook -i hosts 2-linux_config.yml -u admin -e @vars/vars.yml

Execute Oracle deployment playbook.

ansible-playbook -i hosts 4-oracle_config.yml -u admin -e @vars/vars.yml --skip-tags "ora_mount_points,enable_dnfs_client"

Optionally, all above playbooks can be executed from a single playbook run as well.

ansible-playbook -i hosts 0-all_playbook.yml -u admin -e @vars/vars.yml --skip-tags "ora_mount_points,enable_dnfs_client"

Login to EM express to to validate Oracle after successful playbook execution.

Optionally, execute destroy playbook to remove database from DB VM.

ansible-playbook -i hosts 5-destroy.yml -u admin -e @vars/vars.yml