Getting started with the Oracle ZFS Storage VMware Simulator (Part II)

In this Blog post I will share how your newly created Oracle ZFS Storage VMware Simulator can be used to deliver storage for an Oracle 12c database via dNFS and also support HCC (Hybrid Columnar Compression).

Hopefully you have followed Part I of my Blog post on Getting Started with the Oracle ZFS Storage VMware Simulator and already have a working Virtual Machine, if not probably a good time to visit Part I and try to build the simulator.

Oracle dNFS

If you have not set-up or used Oracle dNFS before, you may want to check-out a couple of my previous posts on Oracle dNFS, Using Oracle dNFS Multi-Path and also Oracle dNFS Throughput Testing as these provide some useful background and some additional information.

On my Oracle 12c Linux database server I have created an a directory for the mount point and an oranfstab file in $ORACLE_HOM/dbs thus:

server: zfsa
local: 192.168.56.10  path: 172.16.148.133
nfs_version: nfsv3
export: /export/oradata   mount: /u02/app/oracle/oradata

$ORACLE_HOME/dbs/oranfstab

The parameters used in my oranfstab are described below:

server
The NFS server name, this can be anything and is used for information rather than any network lookups.

local
Up to four paths on the database host, specified by IP address or by name.

path
Up to four network paths to the NFS server, specified either by IP address, or by name.

export
The exported path from the NFS server.

mount
The corresponding local mount point for the exported volume.

nfs_version
Specifies the NFS protocol version used by Direct NFS Client. Possible values are NFSv3NFSv4NFSv4.1, and pNFS. The default version is NFSv3. If you select NFSv4.x, then you must configure the value in oranfstab for nfs_version. Specify nfs_version as pNFS, if you want to use Direct NFS with Parallel NFS.

Verify Oracle dNFS Usage

Restart your Oracle 12c database and view the Oracle Alert file, if you have successfully  configured your environment you should see that Oracle is reporting that the Oracle dNFS library is in use as it’s version.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 4.0

If you are not seeing any reference to the use of Oracle Direct NFS you may need to enable it using the steps detailed here.

We should also be able to see that we are using a single channel, IP addresses and the optional server name from our oranfstab.

Direct NFS: channel id [0] path [172.16.148.133] to filer [zfsa] via local [192.168.56.10] is UP

From the alert log file we can also confirm thatOracle is using our oranfstab file rather than the /etc/mtab, if your oranfstab is not correct your will see defined in mtab rather than defined in oranfstab, time to go back and fix-it.

Direct NFS: attempting to mount /export/oradata on filer zfsa defined in oranfstab

Additionaly, you can query the database using the v$dnfs_servers view to confirm the configuration.

SQL> COLUMN svrname FORMAT A20
SQL> COLUMN dirname FORMAT A50
SQL> SELECT svrname, dirname, nfsversion FROM v$dnfs_servers;

SVRNAME  DIRNAME           NFSVERSION
————— ———————-     ———-
zfsa            /export/oradata NFSv3.0

SNMP (Simple Network Management Protocol)

Finally, before we can start to use Oracle HCC (Hybrid Columnar Compression) within our Oracle 12c database we need to check that our simulator has been configured to use SNMP.

From our Oracle ZFS Storage VMware simulator check the status of services.

SNMP3

And if green, use snmpget to check the response from our simulator, providing the IP address and the Oracle MIB (Management Information Bases) below.

snmpget -v1 -c public 172.16.148.133 1.3.6.1.4.1.42.2.225.1.4.2.0

If all OK, snmpget should return a Sun Storage string, this is required to allow us to use the HCC features which are limited to Oracle Exadata / storage platforms.

SNMPv2-SMI::enterprises.42.2.225.1.4.2.0 = STRING: “Sun Storage 7000”

If you have successfully built your simulator and configured your Oracle 12c environment you are now ready to try to try out some of the HCC features.

In the next Blog post we will attempt to create database objects within and outside of our Oracle ZFS Storage VMware simulator to check our Oracle 12c database is able to use HCC data.

Advertisements

Getting started with the Oracle ZFS Storage VMware Simulator (Part I)

If your like me, you may sometimes need to do some research work with Oracle HCC (Hybrid Columnar Compression), but don’t always have access to an Oracle Exadata machine or ZFS Appliance, then the ZFSA simulator may be your saviour.

On my Mac I use both VirtualBox and VMware Fusion, but for this Blog post I am going to be using VMware Fusion 8.5.10.

Downloading the OVA

You can download the Oracle Storage Appliance Simulator software by following the link  here

ZFS Storage Appliance Home

Once you have accepted the licence agreement you will have the option to download either the VirtualBox or VMware version

downloads

Once the downloads is complete you should have an OS8.7VMware.ova file available to use.

Installing the Virtual Machine

From VMware Fusion navigate to File -> Import and select you OVA file using the ‘Choose File…’ finder.

OVA

Give the VM a name and start the import.

Importing

Once imported you will be presented with a summary screen, if you have spare memory you could consider increasing the allocated memory to 4GB by clicking on the ‘Customise Settings’ otherwise click ‘Finish’

Finish

Boot

Watch the console whilst the VM boots and once up provide a Host Name, DNS Domain name, IP Address, IP Netmask, Default Router and DNS Server as well as a Password for your ‘root’ Super-User.

 

zfsa_browser

Once the initial configuration is complete you can use your browser to configure your ZFS Storage Appliance simulator using the link in the console and the password you just provided.

login

Welcome

If all has gone well you should be presented with the Oracle ZFS Storage Virtual Machine ‘Welcome’ page, click ‘START’ to begin the configuration.

Welcome.pngStep 1

Hit ‘Commit’ to use default network configuration.

Step1

Step 2

Click ‘COMMIT’ to use the DNS values provide during the initial boot.

Step2

Step 3

Again, hit ‘COMMIT’ to use the default NTP settings.

Step3

Step 4

And finally, hit ‘COMMIT’, to take the default Name Services.

Step4

Step 5

We now set up out Storage Pool by clicking on the ‘+’

Step5

PoolName

allocate1

Select the most appropriate Storage Profile for your use, as I will only be using this on my MacBook Pro I will select ‘Striped’ as data protection is not a concern, but I want maximum Performance and Capacity.

storageProfile

The ZFS Appliance will now present a Storage Profile summary screen.

Step5_1

Registration & Support

Enter your MOS details or click ‘LATER’ if you prefer, and confirm.

Support

We can now configure any required Services, LUNS and / or filesystems

Summary

NFS Filesystem

FileSystems

Click on the ‘+’ to create a new Filesystem (NFS Share), provide Name, User, Group and Permissions. e.g.

oradata

Shares2

As I am going to be using this share for my Oracle 12c database I need to configure the storage to use the same block size as my database.

Select the ‘General’ tab to gain access the ‘default’ Project settings, and update the Database record size to 8K in the inherited properties section.

General

SNMP (Simple Network Management Profile)

As I plan to use the ZFS Storage VM to allow my Oracle 12c database to use HCC (Hybrid Columnar Compression) I will configure SNMP and Enable the service.

This can be found hidden away under ‘Configuration’ -> ‘Services’

SysServices

SNMP1

The System Service should now show the SNMP service as Online.

SNMP3

In Part II I will share how our newly created NFS filesystem can be mounted on a Linux client via Oracle dNFS and used by an Oracle 12c database allowing HCC data management.

 

 

 

Oracle dNFS Throughput testing

Oracle Direct NFS

In my previous Oracle dNFS Blog post I detailed how to configure Oracle 12cR2 to use Oracle dNFS multi-path, in this Blog I will share some test results that I achieved in my Oracle 12cR2 Pure Storage FlashBlade lab.

Oracle dNFS Flashblade configuration

Theoretical Limits

A Gigabit Ethernet network can transmit 1,000,000,000 bits per second, as we normally talk about X Bytes per second we need to divide this by 8 as there are 8 bits in a byte.

1GbE = (1,000,000,000 / 8) = 125,000,000 bytes/second
=125,000KB/s
=125MB/s
=0.125GB/s

Unfortunately we can not use all 125,000,000 bytes/second to send data as we have multiple layers of overhead,  when using a default MTU of 1500 we can expect to experience ~6% overhead, this gives us:

1GbE= (1,000,000,000 / 8) less 6% overhead = 117,500,000 bytes/second
=117,500KB/s
=117.5MB/s
=0.118GB/s

1 x  1GbE =   0.118GB/s
1 x 10GbE = 1.175GB/s
4 x 10GbE = 4.7GB/s
8 x 10GbE = 9.4GB/s  (2 Node RAC Cluster)

The dNFS Throughput Test

In the tests below I used Kevin Closson’s excellent SLOB with a custom configuration to produce Oracle FTS (Full Table Scans), many thanks to Frits Hoogland for sharing. This set-up allowed me to generate sufficient load to utilise the available bandwidth and see how close I can get to the ‘theoretical limits’.

Test 1 – Oracle 12cR2 dNFS disabled

OK, during this test we can see that Oracle EM Express is showing a single instance driving ~1.08GB/s IO Throughput

Oracle Express 12c
The Flashblade reporting throughput at ~1.09GB/s

FB_PSTG1

Test 2 – Oracle 12cR2 dNFS enabled

For out next test, I have enabled dNFS with make -f ins_rdbms.mk dnfs_on but have not activated multipath, this was confirmed with the SQL below.

SELECT svrname, dirname, wtmax, rtmax FROM v$dnfs_servers;

SVRNAME        DIRNAME             WTMAX  RTMAX
-------        ------------------  ------ ------
192.168.4.100 /z-fbhosts_oradata   524288 524288
192.168.4.100 /z-fbhosts_orafra    524288 524288
192.168.4.100 /z-fbhosts_oraconfig 524288 524288

Oracle Express EM reports almost identical results with and without dNFS
FB_PSTG1_dnfs
And the Flashblade dashboard reports a similar result.

FB_dnfs
So lets enable multi-path and see the impact.

Test 3 – Oracle 12cR2 dNFS multi-path

So, lets install our oranfstab file and bounce the database and check again using the previous SQL script.

SELECT svrname, dirname, wtmax, rtmax FROM v$dnfs_servers; 
SVRNAME        DIRNAME          WTMAX  RTMAX 
-------        ---------------- ------ ------ 
flashblade /z-fbhosts_oradata   524288 524288 
flashblade /z-fbhosts_orafra    524288 524288 
flashblade /z-fbhosts_oraconfig 524288 524288

We can see the Server Name has been picked up from the oranfstab rather than the mtab, we can also check that we are using multipath with the v$dnfs_channels view.

SELECT distinct svrname, path FROM v$dnfs_channels;
SVRNAME              PATH                                            
-------------------- -------------
flashblade           192.168.7.100
flashblade           192.168.4.100
flashblade           192.168.6.100
flashblade           192.168.5.100

Oracle EM Express is now showing we are using all 4 x 10GbE interfaces as expected and driving 4.03GB/s, also note the increase in Host CPU utilisation.
FB_PSTG1_MP

FB_MP
Ok, our final test will be to see if we get similar results with Oracle RAC

Test 4 – Oracle 12cR2 2 node RAC with dNFS multi-path

For this Oracle RAC test the slob.conf SQLNET_SERVICE_MAX parameter has been changed to 2 and to distribute the workload across the 2 nodes.

FB_PSTG_RAC
Oracle EM Express is showing that the workload has been evenly balanced across the 2 RAC nodes, achieving impressive throughput figures exceeding 8.3GB/s.

FB_RACThe Flashblade dashboard show that the 7 blades were able to deliver a consitent bandwidth of over 8.8GB/s for the Oracle workload utilising all 8 x 10GbE interfaces getting close to wire speed.

In my next Blog Post I plan to enable Jumbo Frames (JF) end-to-end and repeat the above tests.  With JF (MTU 9000) configured, I am hoping to see a ~5% improvement in bandwidth utilisation.


Using Oracle dNFS Multi-path

Oracle Direct NFS (dNFS) Multi-path

If you have multiple interfaces on your database server and your NFS server also supports multiple interfaces you should consider using Oracle dNFS multi-path.

Oracle dNFS multi-path removes the need to distribute and manage datafiles over multiple NFS mount points, Oracle dNFS multi-path can also improve throughput if you were not previously using all your available network interfaces and bandwidth.

In Oracle dNFS (Direct NFS) is disabled by default but can be easily enabled or disabled by following the steps below.

However, before we go through the Oracle dNFS configuration I will describe my lab set-up. My Database Server is configured with 4 x 10GbE all all on different sub-nets, and my  Pure Storage FlashBlade is configured with 4 NFS interfaces exporting 3 NFS mounts for Oracle data, Oracle FRA and shared configuration for my Oracle 12cR2 RAC cluster.

For further details see MOS Note: How to configure DNFS to use multiple IPs (Doc ID 1552831.1). It’s also possible to use the same subnet if required, checkout MOS Note: How to Setup Direct NFS client multipaths in same subnet (Doc ID 822481.1)

dNFS

The oranfstab file

To use Oracle dNFS MultiPath you will need an oranfstab file, Oracle will look in the following order and locations for the configuration file.

  • /etc/oranfstab – Server wide
  • $ORACLE_HOME/dbs – Oracle Home specific

If no oranfstab file is found Oracle will use the /etc/mtab file to enable dNFS but you will have no MultiPath

Below is a copy of my $ORACLE_HOME/dbs/oranfstab

server: flashblade
local: 192.168.4.2 path: 192.168.4.100
local: 192.168.5.2 path: 192.168.5.100
local: 192.168.6.2 path: 192.168.6.100
local: 192.168.7.2 path: 192.168.7.100
nfs_version: nfsv3
export: /z-fbhosts_oradata  mount: /u01/oradata
export: /z-fbhosts_orafra   mount: /u01/app/oracle/fast_recovery_area
export: /z-fbhosts_oraconfig mount: /u01/shared_config

The parameters used in my oranfstab are described below:

server
The NFS server name, this can be anything and is used for information rather than any network lookups.

local
Up to four paths on the database host, specified by IP address or by name.

path
Up to four network paths to the NFS server, specified either by IP address, or by name.

export
The exported path from the NFS server.

mount
The corresponding local mount point for the exported volume.

nfs_version
Specifies the NFS protocol version used by Direct NFS Client. Possible values are NFSv3, NFSv4, NFSv4.1, and pNFS. The default version is NFSv3. If you select NFSv4.x, then you must configure the value in oranfstab for nfs_version. Specify nfs_version as pNFS, if you want to use Direct NFS with Parallel NFS.

Enabling dNFS

Change dir to $ORACLE_HOME/rdbms/lib and use the make command e.g
 cd $ORACLE_HOME/rdbms/lib
  make -f ins_rdbms.mk dnfs_on

You should see something similar to this response:

rm -f /u01/app/oracle/product/12.2.0.1/db_1/rdbms/lib/odm/libnfsodm12.so; \
  cp /u01/app/oracle/product/12.2.0.1/db_1/lib/libnfsodm12.so /u01/app/oracle/product/12.2.0.1/db_1/rdbms/lib/odm/libnfsodm12.so

Verify Oracle dNFS Usage

After applying changes restart the database and confirm your configuration change have had the desired impact.

The Oracle Alert file should now report the Oracle dNFS library is in use as it’s version.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 4.0

We can also see that Oracle is using multiple paths e.g.

Direct NFS: channel id [0] path [192.168.4.100] to filer [flashblade] via local [192.168.4.2] is UP
Direct NFS: channel id [1] path [192.168.5.100] to filer [flashblade] via local [192.168.5.2] is UP
Direct NFS: channel id [2] path [192.168.6.100] to filer [flashblade] via local [192.168.6.2] is UP
Direct NFS: channel id [3] path [192.168.7.100] to filer [flashblade] via local [192.168.7.2] is UP

have had the desired impact by querying the v$dnfs_servers view.

Check to see if Enabled

SQL> COLUMN svrname FORMAT A20
SQL> COLUMN dirname FORMAT A50
SQL> SELECT svrname, dirname, nfsversion FROM v$dnfs_servers;

SVRNAME     DIRNAME                                     NFSVERSION
-------------------- -----------------------------------------------
flashblade     /z-fbhosts_oraconfig                       NFSv3.0
flashblade     /z-fbhosts_oradata                         NFSv3.0
flashblade     /z-fbhosts_orafra                          NFSv3.0

dNFS views

  • v$dnfs_stats          – shows dNFS performance statistics
  • v$dnfs_channels  – shows network channels is use by dNFS
  • v$dnfs_servers     – shows dNFS servers
  • v$dnfs_files           – shows files open for dNFS

Disabling dNFS

To disable dNFS change directory to $ORACLE_HOME/rdbms/lib and enter the following commands:

cd $ORACLE_HOME/rdbms/lib
  make -f ins_rdbms.mk dnfs_off

You should see something similar to this response:

rm -f /u01/app/oracle/product/12.2.0.1/db_1/rdbms/lib/odm/libnfsodm12.so

You can also now remove the previously created oranfstab file

Check to see if Disabled

SQL> COLUMN svrname FORMAT A20
SQL> COLUMN dirname FORMAT A50
SQL> SELECT svrname, dirname, nfsversion FROM v$dnfs_servers;
 
no rows selected

In my next post I will share some before and after test results.