TR-4986: Simplified, Automated Oracle Deployment on Amazon FSx ONTAP with iSCSI
Allen Cao, Niyaz Mohamed, NetApp
This solution provides overview and details for automated Oracle deployment and protection in Amazon FSx ONTAP as primary database storage with iSCSI protocol and Oracle database configured in standalone ReStart using Oracle asm as volume manager.
Purpose
Amazon FSx ONTAP is a storage service that allows you to launch and run fully managed NetApp ONTAP file systems in the AWS Cloud. It provides the familiar features, performance, capabilities, and APIs of NetApp file systems with the agility, scalability, and simplicity of a fully managed AWS service. It empowers you to run the most demanding database workload, such as Oracle, in the AWS cloud with peace of mind.
This documentation demonstrates the simplified deployment of Oracle databases in an Amazon FSx ONTAP file system using Ansible automation. The Oracle database is deployed in a standalone ReStart configuration with iSCSI protocol for data access and Oracle ASM for database storage disks management. It also provides information on Oracle database backup, restore, and clone using the NetApp SnapCenter UI tool for storage-efficient database operation in AWS Cloud.
This solution addresses the following use cases:
-
Automated Oracle database deployment on Amazon FSx ONTAP file system
-
Oracle database backup and restore on Amazon FSx ONTAP file system using NetApp SnapCenter tool
-
Oracle database clone for dev/test or other use cases on Amazon FSx ONTAP file system using NetApp SnapCenter tool
Audience
This solution is intended for the following people:
-
A DBA who would like to deploy Oracle on Amazon FSx ONTAP file system.
-
A database solution architect who would like to test Oracle workloads on Amazon FSx ONTAP file system.
-
A storage administrator who would like to deploy and manage an Oracle database on Amazon FSx ONTAP file system.
-
An application owner who would like to stand up an Oracle database on Amazon FSx ONTAP file system.
Solution test and validation environment
The testing and validation of this solution were performed in a lab setting that might not match the final deployment environment. See the section Key factors for deployment consideration for more information.
Architecture
Hardware and software components
Hardware |
||
Amazon FSx ONTAP storage |
Current version offered by AWS |
One FSx HA cluster in the same VPC and availability zone |
EC2 instance for compute |
t2.xlarge/4vCPU/16G |
Two EC2 T2 xlarge EC2 instances for concurrent deployment |
Software |
||
RedHat Linux |
RHEL-8.6, 4.18.0-372.9.1.el8.x86_64 kernel |
Deployed RedHat subscription for testing |
Windows Server |
2022 Standard, 10.0.20348 Build 20348 |
Hosting SnapCenter server |
Oracle Grid Infrastructure |
Version 19.18 |
Applied RU patch p34762026_190000_Linux-x86-64.zip |
Oracle Database |
Version 19.18 |
Applied RU patch p34765931_190000_Linux-x86-64.zip |
Oracle OPatch |
Version 12.2.0.1.36 |
Latest patch p6880880_190000_Linux-x86-64.zip |
SnapCenter Server |
Version 4.9P1 |
Workgroup deployment |
Open JDK |
Version java-1.8.0-openjdk.x86_64 |
SnapCenter plugin requirement on DB VMs |
Oracle database configuration in the lab environment
Server |
Database |
DB Storage |
ora_01 |
NTAP1(NTAP1_PDB1,NTAP1_PDB2,NTAP1_PDB3) |
iSCSI luns on Amazon FSx ONTAP file system |
ora_02 |
NTAP2(NTAP2_PDB1,NTAP2_PDB2,NTAP2_PDB3) |
iSCSI luns on Amazon FSx ONTAP file system |
Key factors for deployment consideration
-
Oracle database storage layout. In this automated Oracle deployment, we provision four database volumes to host Oracle binary, data, and logs by default. A single lun in a volume allocates to Oracle binary. We then create two ASM disk groups from data and logs luns. Within the +DATA asm disk group, we provision two data volumes with two luns in a volume. Within the +LOGS asm disk group, we create two luns in a log volume. Multiple luns laid out within an ONTAP volume provides better performance in general.
-
Multiple DB servers deployment. The automation solution can deploy an Oracle container database to multiple DB servers in a single Ansible playbook run. Regardless of the number of DB servers, the playbook execution remains the same. You can deploy multiple container databases to a single EC2 instance with different database instance IDs (Oracle SID). But ensure there is sufficient memory on the host to support deployed databases.
-
iSCSI configuration. The EC2 instance database server connects to FSx storage with the iSCSI protocol. EC2 instances generally deploy with a single network interface or ENI. The single NIC interface carries both iSCSI and application traffic. It is important to gauge the Oracle database peak I/O throughput requirement by carefully analyzing the Oracle AWR report in order to choose the right EC2 compute instance that meets both application and iSCSI traffic-throughput requirements. Also, AWS EC2 generally limits each TCP flow to 5 Gbps. Each iSCSI path provides 5 Gbps (625 MBps) of bandwidth, and multiple iSCSI connections may be required to support higher throughput requirements.
-
Oracle ASM redundancy level to use for each Oracle ASM disk group that you create. Because the Amazon FSx ONTAP is HA enabled for data protection at the cluster disk level, you should use
External Redundancy
, which means that the option does not allow Oracle ASM to mirror the contents of the disk group. -
Database backup. NetApp provides a SnapCenter software suite for database backup, restore, and cloning with a user-friendly UI interface. NetApp recommends implementing such a management tool to achieve fast (under a minute) SnapShot backup, quick (minutes) database restore, and database clone.
Solution deployment
The following sections provide step-by-step procedures for automated Oracle 19c deployment and protection on Amazon FSx ONTAP file system with directly mounted database luns via iSCSI to EC2 instance VM in a single node Restart configuration with Oracle ASM as database volume manager.
Prerequisites for deployment
Details
Deployment requires the following prerequisites.
-
An AWS account has been set up, and the necessary VPC and network segments have been created within your AWS account.
-
From the AWS EC2 console, deploy EC2 Linux instances as Oracle DB servers. Enable SSH private/public key authentication for ec2-user. See the architecture diagram in the previous section for details about the environment setup. Also review the User Guide for Linux instances for more information.
-
From the AWS FSx console, provision an Amazon FSx ONTAP file system that meets the requirements. Review the documentation Creating FSx ONTAP file systems for step-by-step instructions.
-
Steps 2 and 3 can be performed using the following Terraform automation toolkit, which creates an EC2 instance named
ora_01
and an FSx file system namedfsx_01
. Review the instruction carefully and change the variables to suit your environment before execution. The template can be easily revised for your own deployment requirements.git clone https://github.com/NetApp-Automation/na_aws_fsx_ec2_deploy.git
-
Provision an EC2 Linux instance as the Ansible controller node with the latest version of Ansible and Git installed. Refer to the following link for details: Getting Started with NetApp solution automation in section -
Setup the Ansible Control Node for CLI deployments on RHEL / CentOS
or
Setup the Ansible Control Node for CLI deployments on Ubuntu / Debian
. -
Provision a Windows server to run the NetApp SnapCenter UI tool with the latest version. Refer to the following link for details: Install the SnapCenter Server
-
Clone a copy of the NetApp Oracle deployment automation toolkit for iSCSI.
git clone https://bitbucket.ngage.netapp.com/scm/ns-bb/na_oracle_deploy_iscsi.git
-
Stage following Oracle 19c installation files on EC2 instances /tmp/archive directory.
installer_archives: - "LINUX.X64_193000_grid_home.zip" - "p34762026_190000_Linux-x86-64.zip" - "LINUX.X64_193000_db_home.zip" - "p34765931_190000_Linux-x86-64.zip" - "p6880880_190000_Linux-x86-64.zip"
Ensure that you have allocated at least 50G in Oracle VM root volume to have sufficient space to stage Oracle installation files. -
Watch the following video:
Simplified and automated Oracle deployment on Amazon FSx ONTAP with iSCSI
Automation parameter files
Details
Ansible playbook executes database installation and configuration tasks with predefined parameters. For this Oracle automation solution, there are three user-defined parameter files that need user input before playbook execution.
-
hosts - define targets that the automation playbook is running against.
-
vars/vars.yml - the global variable file that defines variables that apply to all targets.
-
host_vars/host_name.yml - the local variable file that defines variables that apply only to a named target. In our use case, these are the Oracle DB servers.
In addition to these user-defined variable files, there are several default variable files that contain default parameters that do not require change unless necessary. The following sections show how to configure the user-defined variable files.
Parameter files configuration
Details
-
Ansible target
hosts
file configuration:# Enter Amazon FSx ONTAP management IP address [ontap] 172.16.9.32 # Enter name for ec2 instance (not default IP address naming) to be deployed one by one, follow by ec2 instance IP address, and ssh private key of ec2-user for the instance. [oracle] ora_01 ansible_host=10.61.180.21 ansible_ssh_private_key_file=ora_01.pem ora_02 ansible_host=10.61.180.23 ansible_ssh_private_key_file=ora_02.pem
-
Global
vars/vars.yml
file configuration############################################################################################################# ###### Oracle 19c deployment global user configurable variables ###### ###### Consolidate all variables from ONTAP, linux and oracle ###### ############################################################################################################# ############################################################################################################# ###### ONTAP env specific config variables ###### ############################################################################################################# # Enter the supported ONTAP platform: on-prem, aws-fsx. ontap_platform: aws-fsx # Enter ONTAP cluster management user credentials username: "fsxadmin" password: "xxxxxxxx" ############################################################################################################# ### Linux env specific config variables ### ############################################################################################################# # Enter RHEL subscription to enable repo redhat_sub_username: xxxxxxxx redhat_sub_password: "xxxxxxxx" ############################################################################################################# ### Oracle DB env specific config variables ### ############################################################################################################# # Enter Database domain name db_domain: solutions.netapp.com # Enter initial password for all required Oracle passwords. Change them after installation. initial_pwd_all: xxxxxxxx
-
Local DB server
host_vars/host_name.yml
configuration such as ora_01.yml, ora_02.yml …# User configurable Oracle host specific parameters # Enter container database SID. By default, a container DB is created with 3 PDBs within the CDB oracle_sid: NTAP1 # Enter database shared memory size or SGA. CDB is created with SGA at 75% of memory_limit, MB. The grand total of SGA should not exceed 75% available RAM on node. memory_limit: 8192
Playbook execution
Details
There are a total of six playbooks in the automation toolkit. Each performs different task blocks and serves different purposes.
0-all_playbook.yml - execute playbooks from 1-4 in one playbook run. 1-ansible_requirements.yml - set up Ansible controller with required libs and collections. 2-linux_config.yml - execute Linux kernel configuration on Oracle DB servers. 3-ontap_config.yml - configure ONTAP svm/volumes/luns for Oracle database and grant DB server access to luns. 4-oracle_config.yml - install and configure Oracle on DB servers for grid infrastructure and create a container database. 5-destroy.yml - optional to undo the environment to dismantle all.
There are three options to run the playbooks with the following commands.
-
Execute all deployment playbooks in one combined run.
ansible-playbook -i hosts 0-all_playbook.yml -u ec2-user -e @vars/vars.yml
-
Execute playbooks one at a time with the number sequence from 1-4.
ansible-playbook -i hosts 1-ansible_requirements.yml -u ec2-user -e @vars/vars.yml
ansible-playbook -i hosts 2-linux_config.yml -u ec2-user -e @vars/vars.yml
ansible-playbook -i hosts 3-ontap_config.yml -u ec2-user -e @vars/vars.yml
ansible-playbook -i hosts 4-oracle_config.yml -u ec2-user -e @vars/vars.yml
-
Execute 0-all_playbook.yml with a tag.
ansible-playbook -i hosts 0-all_playbook.yml -u ec2-user -e @vars/vars.yml -t ansible_requirements
ansible-playbook -i hosts 0-all_playbook.yml -u ec2-user -e @vars/vars.yml -t linux_config
ansible-playbook -i hosts 0-all_playbook.yml -u ec2-user -e @vars/vars.yml -t ontap_config
ansible-playbook -i hosts 0-all_playbook.yml -u ec2-user -e @vars/vars.yml -t oracle_config
-
Undo the environment
ansible-playbook -i hosts 5-destroy.yml -u ec2-user -e @vars/vars.yml
Post execution validation
Details
After the playbook run, login to the Oracle DB server as oracle user to validate that Oracle grid infrastructure and database are created successfully. Following is an example of Oracle database validation on host ora_01.
-
Validate Oracle container database on EC2 instance
[admin@ansiblectl na_oracle_deploy_iscsi]$ ssh -i ora_01.pem ec2-user@172.30.15.40 Last login: Fri Dec 8 17:14:21 2023 from 10.61.180.18 [ec2-user@ip-172-30-15-40 ~]$ uname -a Linux ip-172-30-15-40.ec2.internal 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux [ec2-user@ip-172-30-15-40 ~]$ sudo su [root@ip-172-30-15-40 ec2-user]# su - oracle Last login: Fri Dec 8 16:25:52 UTC 2023 on pts/0 [oracle@ip-172-30-15-40 ~]$ sqlplus / as sysdba SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 8 18:18:20 2023 Version 19.18.0.0.0 Copyright (c) 1982, 2022, Oracle. All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.18.0.0.0 SQL> select name, open_mode, log_mode from v$database; NAME OPEN_MODE LOG_MODE --------- -------------------- ------------ NTAP1 READ WRITE ARCHIVELOG SQL> show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 NTAP1_PDB1 READ WRITE NO 4 NTAP1_PDB2 READ WRITE NO 5 NTAP1_PDB3 READ WRITE NO SQL> select name from v$datafile; NAME -------------------------------------------------------------------------------- +DATA/NTAP1/DATAFILE/system.257.1155055419 +DATA/NTAP1/DATAFILE/sysaux.258.1155055463 +DATA/NTAP1/DATAFILE/undotbs1.259.1155055489 +DATA/NTAP1/86B637B62FE07A65E053F706E80A27CA/DATAFILE/system.266.1155056241 +DATA/NTAP1/86B637B62FE07A65E053F706E80A27CA/DATAFILE/sysaux.267.1155056241 +DATA/NTAP1/DATAFILE/users.260.1155055489 +DATA/NTAP1/86B637B62FE07A65E053F706E80A27CA/DATAFILE/undotbs1.268.1155056241 +DATA/NTAP1/0C03AAFA7C6FD2E5E063280F1EACFBE0/DATAFILE/system.272.1155057059 +DATA/NTAP1/0C03AAFA7C6FD2E5E063280F1EACFBE0/DATAFILE/sysaux.273.1155057059 +DATA/NTAP1/0C03AAFA7C6FD2E5E063280F1EACFBE0/DATAFILE/undotbs1.271.1155057059 +DATA/NTAP1/0C03AAFA7C6FD2E5E063280F1EACFBE0/DATAFILE/users.275.1155057075 NAME -------------------------------------------------------------------------------- +DATA/NTAP1/0C03AC0089ACD352E063280F1EAC12BD/DATAFILE/system.277.1155057075 +DATA/NTAP1/0C03AC0089ACD352E063280F1EAC12BD/DATAFILE/sysaux.278.1155057075 +DATA/NTAP1/0C03AC0089ACD352E063280F1EAC12BD/DATAFILE/undotbs1.276.1155057075 +DATA/NTAP1/0C03AC0089ACD352E063280F1EAC12BD/DATAFILE/users.280.1155057091 +DATA/NTAP1/0C03ACEABA54D386E063280F1EACE573/DATAFILE/system.282.1155057091 +DATA/NTAP1/0C03ACEABA54D386E063280F1EACE573/DATAFILE/sysaux.283.1155057091 +DATA/NTAP1/0C03ACEABA54D386E063280F1EACE573/DATAFILE/undotbs1.281.1155057091 +DATA/NTAP1/0C03ACEABA54D386E063280F1EACE573/DATAFILE/users.285.1155057105 19 rows selected. SQL> select name from v$controlfile; NAME -------------------------------------------------------------------------------- +DATA/NTAP1/CONTROLFILE/current.261.1155055529 +LOGS/NTAP1/CONTROLFILE/current.256.1155055529 SQL> select member from v$logfile; MEMBER -------------------------------------------------------------------------------- +DATA/NTAP1/ONLINELOG/group_3.264.1155055531 +LOGS/NTAP1/ONLINELOG/group_3.259.1155055539 +DATA/NTAP1/ONLINELOG/group_2.263.1155055531 +LOGS/NTAP1/ONLINELOG/group_2.257.1155055539 +DATA/NTAP1/ONLINELOG/group_1.262.1155055531 +LOGS/NTAP1/ONLINELOG/group_1.258.1155055539 6 rows selected. SQL> exit Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.18.0.0.0
-
Validate Oracle listener.
[oracle@ip-172-30-15-40 ~]$ lsnrctl status listener LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 08-DEC-2023 18:20:24 Copyright (c) 1991, 2022, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-172-30-15-40.ec2.internal)(PORT=1521))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production Start Date 08-DEC-2023 16:26:09 Uptime 0 days 1 hr. 54 min. 14 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/oracle/product/19.0.0/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/ip-172-30-15-40/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ip-172-30-15-40.ec2.internal)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=ip-172-30-15-40.ec2.internal)(PORT=5500))(Security=(my_wallet_directory=/u01/app/oracle/product/19.0.0/NTAP1/admin/NTAP1/xdb_wallet))(Presentation=HTTP)(Session=RAW)) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM", status READY, has 1 handler(s) for this service... Service "+ASM_DATA" has 1 instance(s). Instance "+ASM", status READY, has 1 handler(s) for this service... Service "+ASM_LOGS" has 1 instance(s). Instance "+ASM", status READY, has 1 handler(s) for this service... Service "0c03aafa7c6fd2e5e063280f1eacfbe0.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... Service "0c03ac0089acd352e063280f1eac12bd.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... Service "0c03aceaba54d386e063280f1eace573.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... Service "NTAP1.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... Service "NTAP1XDB.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... Service "ntap1_pdb1.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... Service "ntap1_pdb2.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... Service "ntap1_pdb3.solutions.netapp.com" has 1 instance(s). Instance "NTAP1", status READY, has 1 handler(s) for this service... The command completed successfully
-
Validate the grid infrastructure and resources created.
[oracle@ip-172-30-15-40 ~]$ asm [oracle@ip-172-30-15-40 ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [oracle@ip-172-30-15-40 ~]$ crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE ip-172-30-15-40 STABLE ora.LISTENER.lsnr ONLINE ONLINE ip-172-30-15-40 STABLE ora.LOGS.dg ONLINE ONLINE ip-172-30-15-40 STABLE ora.asm ONLINE ONLINE ip-172-30-15-40 Started,STABLE ora.ons OFFLINE OFFLINE ip-172-30-15-40 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE ONLINE ip-172-30-15-40 STABLE ora.diskmon 1 OFFLINE OFFLINE STABLE ora.driver.afd 1 ONLINE ONLINE ip-172-30-15-40 STABLE ora.evmd 1 ONLINE ONLINE ip-172-30-15-40 STABLE ora.ntap1.db 1 ONLINE ONLINE ip-172-30-15-40 Open,HOME=/u01/app/o racle/product/19.0.0 /NTAP1,STABLE --------------------------------------------------------------------------------
-
Validate Oracle ASM.
[oracle@ip-172-30-15-40 ~]$ asmcmd ASMCMD> lsdg State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 512 4096 4194304 163840 155376 0 155376 0 N DATA/ MOUNTED EXTERN N 512 512 4096 4194304 81920 80972 0 80972 0 N LOGS/ ASMCMD> lsdsk Path AFD:ORA_01_DAT1_01 AFD:ORA_01_DAT1_03 AFD:ORA_01_DAT2_02 AFD:ORA_01_DAT2_04 AFD:ORA_01_LOGS_01 AFD:ORA_01_LOGS_02 ASMCMD> afd_state ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'ip-172-30-15-40.ec2.internal' ASMCMD> exit
-
Login to Oracle Enterprise Manager Express to validate database.
Oracle backup, restore, and clone with SnapCenter
Details
Refer to TR-4979 Simplified, self-managed Oracle in VMware Cloud on AWS with guest-mounted FSx ONTAP section Oracle backup, restore, and clone with SnapCenter
for details on setting up SnapCenter and executing the database backup, restore, and clone workflows.
Where to find additional information
To learn more about the information described in this document, review the following documents and/or websites:
-
Amazon FSx ONTAP
-
Amazon EC2
-
Installing Oracle Grid Infrastructure for a Standalone Server with a New Database Installation
-
Installing and Configuring Oracle Database Using Response Files
-
Use Red Hat Enterprise Linux 8.2 with ONTAP