Oracle VM 3.4.6 : Part 6 – OVM Block Storage

Oracle VM Block Storage

In my last Blog post I presented NFS storage to my Oracle VM Server, in this post I will show how block storage can also be configured for use with OVM.

Create Pure FlashArray volume

Our first step is to create a volume and connect it to our Oracle VM Server. In this example I have created a 1TB volume on one of my lab Pure Storage FlashArrays and connected it to my OVM Server.

Create Volume
Volume details
Connected Hosts

Configure OVM – SAN Servers

Before we present our new Pure Storage block device to our Oracle VM Server we first need to update the /etc/multipath.conf file to include any vendor specific settings and restart the multipath service. Below is the entry for a Pure Storage FlashArray.

        device {
                vendor                "PURE"
                product               "FlashArray"
                path_selector         "round-robin 0"
                path_grouping_policy  group_by_prio
                path_checker          tur
                fast_io_fail_tmo      10
                dev_loss_tmo          60
                no_path_retry         0
                hardware_handler      "1 alua"
                prio                  alua
                failback              immediate
                user_friendly_names   no
        }
[root@z-ovm ~]# service multipathd reload

Now Logon to your Oracle VM Manager and refresh your previously created SAN Server to discover our newly created volume(s).

Refresh SAN Servers

Click OK to confirm Refresh, the new LUN(s) should now be visible in Oracle VM Manager Storage tab.

As I have presented the block storage over iSCSI we can see IQN (iSCSI Qualified Name) in the Storage Targets, this was set-up in Part 4 OVM Storage.

ID: is the UUID is a universally unique identifier that Oracle VM Manager assigns to a physical disk.

Page83 ID: The unique SCSI identifier for the physical disk, for a Pure Storage FlashArray this will be set as the Vendor ID + lowercase volume serial number e.g 3624a9370 + 513519106E354B37002EB1D1.

NOTE: Oracle VM does not currently support User Friendly Names (see extract from /etc/multipath.conf.

## IMPORTANT for OVS this must be no. OVS does not support user friendly
## names and instead uses the WWIDs as names.
defaults {
        user_friendly_names no
        getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
        no_path_retry 10
}

Therefore the User Friendly Name will be same as the Page83 ID.

An optional step now is to Right Click the volume, and select Edit to provide a more meaningful Name and Description. Once updated Right Click again and this time select Refresh.

Updated volume names

Create Oracle VM Repository

From the Repository tab click on the Green Plus to create a new Repository, enter a Repository Name, select Physical Disk, Description, for non-Clustered deployments select Server Pool None and click on the Magnifying Glass to select the Physical Disk previously added. Then click Next.

Select SAN Server and Name, check the User Friendly Name is as expected.

Create a Repository: Select Physical Disk

Oracle VM Server

If we now logon to the Oracle VM Server as root we should now see our newly created volume(s).

[root@z-ovm ~]# multipath -ll
3624a9370513519106e354b37002eb1d1 dm-1 PURE,FlashArray
size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='queue-length 0' prio=1 status=active
  |- 47:0:0:1 sdb 8:16 active ready running
  |- 48:0:0:1 sdc 8:32 active ready running
  |- 49:0:0:1 sdd 8:48 active ready running
  `- 50:0:0:1 sde 8:64 active ready running
[root@z-ovm ~]# ls -l /dev/mapper/
total 0
lrwxrwxrwx 1 root root       7 Jan  3 15:26 3624a9370513519106e354b37002eb1d1 -> ../dm-1

Each repository contains the following directory structure /OVS/Repositories/*/

  • Assemblies
    • Contains pre-configured sets of virtual machines.
  • ISOs
    • Contains ISO images which can be used by VM’s
  • Templates
    • Contains virtual machine templates
  • VirtualDisks
    • Contains dedicated or shared virtual disks
  • VirtualMachines
    • Contains virtual machine configuration files

However before we logoff our Oracle VM server lets use findmnt to check that our NFS and OCFS2 filesystems mounts look OK. In the example below we can see both our NFS export and block devices as expected.

[root@z-ovm mapper]# findmnt -t nfs
TARGET                                             SOURCE                      FSTYPE OPTIONS
/OVS/Repositories/0004fb00000300009e6780edf21a0ee5 192.168.4.100:/z-ovm_images nfs    rw,relatime,vers=3,rsize=52

[root@z-ovm mapper]# findmnt -t ocfs2
TARGET                                            SOURCE                                       FSTYPE OPTIONS
/OVS/Repositories/0004fb0000030000b72890166aa11f29                                                   /dev/mapper/3624a9370513519106e354b37002eb1d1                                                                                              ocfs2  rw,relatime

You can read-up on OCFS2 at: https://oss.oracle.com/projects/ocfs2/documentation/

Advertisements

Oracle VM 3.4.6 : Part 5 – OVM NFS Storage

Oracle VM NFS Storage

If you been following my series of Blog Post on OVM you should by now have a working OVM Server and Manager. However, if you a new to this series you can catch-up using the links below:

Pure Storage FlashBlade

In this Blog post we will create our first OVM Repository for Oracle VM images using NFS storage.

Our first step is to create an NFS File System and export it to our OVM Server. In the example I am creating a 5TB file system on one of my lab Pure Storage FlashBlades with snapshots enabled and access limited to my OVM Server.

Create File System

Oracle VM Manager File Server

Logon to your Oracle VM Manager and refresh your previously created NFS File Server to discover our newly created NFS export.

Refresh File Server

Click OK to confirm Refresh, our new file system should now be visible in Oracle VM Manager Storage tab.

File System – Default Name

An optional step now is to Right Click the file system, and select Edit to provide a more meaning full name and description. Once updated Right Click again and this time select Refresh.

Updated and refreshed File System

Create Oracle VM Repository

Click on the Green Plus to create a new Repository, enter a Repository Name, default Network File Server, Description and click on the Magnifying Glass to select the Network File System previously added. Then click Next.

Repository Information
Create a Repository: Select Network File System

Select OVM Server and move it to Present to Server(s) and click Finish.

Present to Servers

Oracle VM Server

If we now logon to the Oracle VM Server as root we can now see our newly created NFS mount point, and the Oracle VM created directory structure e.g.

Repository 
- Assemblies
- ISOs
- Templates
- VirtualDisks
- VirtualMachines
z-ovm_images – 5TB FlashBlade File System
Repository directory structure

In my next Blog I will create another Oracle VM repository for my planned Oracle VM VirtualMachine but this time using iSCSI block storage.

Oracle VM 3.4.6 : Part 4 – OVM Storage

Configuring OVM Storage

If you have followed Part 1, Part 2 and Part 3 you should now have a working networked OVM Server and Manager ready to have storage presented to it.

Before you start, logon to the OVM Server and add your NFS and / or iSCSI fully qualified domain named servers to the /etc/host file to avoid hitting OVMRU_000082E.

Can Not Add NFS Server To Oracle VM Manager: OVMRU_000082E Either the hostname or IP address is invalid (Doc ID 2294262.1)

Navigate to the Storage tab, from here we can configure SAN Servers (Block) and File Servers (NFS)

Storage – File Servers

NFS Storage

To add NFS Storage, select File Servers and click the folder icon with the green plus, this should show the Discover File Server.. tag.

Enter NFS Server Name, storage network IP address, Description

File Server Parameters

Move OVM server(s) to Selected Admin Server(s) and click Next.

Add Admin Servers

Move OVM server(s) to Selected Refresh Server(s) and click Next.

Select Refresh Servers

Block Storage (iSCSI)

Storage – SAN Servers

Enter SAN Server Name and Description, as I am planning to use iSCSI there is no need to change Storage Type, click Next.

Check you do not have any leading or trailing spaces in your description, otherwise you will encounter an OVMRU_000096E error.

OVMRU_000096E Cannot set attribute Description of object IscsiStorageArray to value [ xxx ]. Leading or trailing spaces are not allowed.

Enter Access Host(s) and Click Next.

Create Access Host

Repeat above step for multipathing, and then click Next

Access Information

Select OVM Server(s) and click Next

Add Admin Servers

Select the Default access group and click the pencil icon to edit Access Group.

Manage Access Group

Click on the Storage Initiators tab and move the generic Oracle storage initiator to Selected Storage Initiators and click OK.

Edit Access Group

Now click Finish to complete the set-up.

Creating Oracle support diagnostics for OVM 3.4.6

Using vmpinfo3

VMPinfo3 is an Oracle support tool for capturing log files and other details from Oracle VM 3.x Manager and Servers. To use it ssh to you Oracle VM Manager as root and execute vmpinfo3.sh as below:

The Oracle Support Document VMPinfo3 Diagnostic Tool For Oracle VM 3.2, 3.3 and 3.4 Troubleshooting (Doc ID 1521931.1) directs you to /u01/app/oracle/ovm-manager-3/ovm_shell/tools/vmpinfo/ however in 3.4.6 vmpinfo3.sh has been relocated to /u01/app/oracle/ovm-manager-3/ovm_tools/support/

First, start by using the listservers option to report the OVM Servers known by your OVM Manager e.g.

[root@z-ovmm ~]# cd  /u01/app/oracle/ovm-manager-3/ovm_tools/support
[root@z-ovmm support]# ./vmpinfo3.sh --username=admin listservers
Enter OVM Manager Password: 

The following server(s) are owned by this manager: ['z-ovm']

Next, re-run vmpinfo3 this time with the servers option providing the server(s) listed, note if you have multiple OVM servers use a comma delimiter e.g. servers=server1,server2,server3

[root@z-ovmm support]# ./vmpinfo3.sh --username=admin servers=z-ovm
Enter OVM Manager Password: 

Starting data collection
Gathering files from servers: z-ovm. This process may take some time.
Gathering OVM Model Dump files
Gathering sosreports from servers
The following server(s) will get info collected: [z-ovm]
Gathering sosreport from z-ovm
Data collection complete
Gathering OVM Manager Logs
Clean up metrics
Copying model files
Copying DB backup log files
Running lightweight sosreport
Archiving vmpinfo3-20181220-121620
=========================================================================
 Please send /tmp/vmpinfo3-3.4.6.2105-20181220-121620.tar.gz to Oracle support
=========================================================================

You can now upload the compressed tar file to your open SR.

Oracle VM 3.4.6 : Part 3 – OVM Networks

Configuring OVM Networks

Before we can present our iSCSI and NFS storage to our Oracle VM Server we need to configure a network. Oracle VM Manager provides a simple to use interface which can maintain the /etc/sysconfig/networking-scripts without the need of manual editing.

In my lab I have a number of VLAN’s configure on an Arista switch, in this Blog I will using an iSCSI VLAN and my NFS VLAN.

Logon to Oracle VM Manager and select the Networking Tab, then VLAN Interfaces.


Drill down to your OVM server, is this example mine is called z-ovm, here you will be presented with a list of available network interfaces on your selected server. Select your interface and click Next.

Create VLAN

Add your VLAN ID and click Next.

Change Addressing to Static and enter IP Address and Netmask, click Finish to Configure IP Address.

Now navigate to the Networks tab and click the green + to Create a Network entry.

Create Network

In the Blog we will create 2 Network’s one for NFS and the other for iSCSI, enter Name, Description and select Network Uses e.g Storage

As we have already configure a VLAN we can click Next, and go straight to Select VLAN Interfaces.

Select available VLAN and click Next.

Now it click Finish to complete setup.

We can now see our new NFS storage network. 

I repeated the above steps for my iSCSI VLAN and network.

You can validate and review your check network configuration changes by logging to your OVM sever and navigating to the network-scripts directory e.g. 

[root@z-ovm ~]# cd /etc/sysconfig/network-scripts/
[root@z-ovm network-scripts]# 

From here we can see the network files have been updated by Oracle VM Manager and configured to use our VLAN’s

[root@z-ovm network-scripts]# cat ifcfg-eth1.3
#This file was dynamically created by Oracle VM Manager. Please do not edit
DEVICE=eth1.3
HWADDR=EC:F4:BB:C7:9B:F2
BOOTPROTO=static
IPADDR=192.168.2.43
NETMASK=255.255.255.0
ONBOOT=yes
VLAN=yes
[root@z-ovm network-scripts]# ifconfig eth1.3
eth1.3    Link encap:Ethernet  HWaddr EC:F4:BB:C7:9B:F2  
          inet addr:192.168.2.43  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1777 errors:0 dropped:31 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:93802 (91.6 KiB)  TX bytes:672 (672.0 b)


Oracle VM 3.4.6 : Part 1 – OVM Server

Oracle VM Server

Oracle VM is a free to download virtualisation technology and for many deployments could provide a great alternative to VMware. In this series of Blog Posts I am going to give OracleVM a spin, and document my experiences. 

Oracle VM Server is a Bare-Metal install unlike Oracle VM Manager which is installed on an OS. In part 2 of this Blog series I will document that part of the install, if you already have Oracle VM Server running you can jump to Part 2 here.

Oracle VM Server and Oracle VM Manager can both be downloaded from the Oracle Technology Network (OTN) or the Oracle eDelivery site.

Once you have downloaded the 841MB Oracle VM Server ISO (OVS-3.4.6-2105.iso) , note you will need to burn a CD/DVD for a physical server install.

Oracle VM Server Install

Boot from the CD and hit Enter to begin the installation process. It will feel very familiar to anyone who as done a Linux Install and be completed with just a keyboard by using the navigation, tab and enter keys.

Initial boot screen
Hit Enter
Media Validation
Tab to ‘Skip’ and hit Enter
Welcome to Oracle VM Sever
Hit Enter
Language Selection
Select required language, tab to ‘OK’ and hit Enter
Keyboard Selection
Select keyboard layout, tab to ‘OK’ and hit Enter
Oracle VM Server Licence Agreement
If OK tab to ‘Accept’ and hit Enter
Partitioning Type
Review and confirm Disk Partitioning, tab to ‘OK’ and hit Enter
Oracle VM kdump Info
Tab to ‘Yes’ if required, and hit Enter
Oracle VM Management Interface
Configure Network Interfaces, tab to ‘OK’ and hit Enter
IPv4 Configuration for eth0
Configure IP address manually, for lab work you can use DHCP, then tab to ‘OK’ and hit Enter 
Hostname Configuration
Manually configure Hostname, tab to ‘OK’ and hit Enter
Time Zone Selection
Select required Time Zone, tan to ‘OK’ and hit Enter
Oracle VM Agent password
Enter password, confirm then tab to ‘OK’ and hit Enter
Root Password
Enter password, confirm then tab to ‘OK’ and hit Enter
You can now sit back and watch the Package Installation…
Complete
Hit Enter to Reboot
Oracle VM Server 3.4.6 Console
Hit Alt F2 for login console
Login as root using password previously provided

Hopefully you now have a working Oracle VM Server, if so you may want to continue to Part 2 to have a look at my Oracle VM Manager installation walkthrough.

Oracle VM 3.4.6 :Part 2 – OVM Manager

Oracle VM Manager

Oracle VM is a free to download virtualisation technology and for many deployments could provide a great alternative to VMware. In this series of Blog Posts I am going to give OracleVM a spin, and document my experiences. 

Oracle VM Manager needs to be installed into an existing OS unlike Oracle VM Server (see Part 1 of this series) which is a Bare-Metal install. For this guide I will be using an Oracle Linux 7.6 and a loopback mount to the OracleVM Manager 3.4 ISO, rather than a physical DVD. Before you start downloaded the ISO from edelivery or OTN e.g. Oracle Technology Network / Server and Storage Systems / Oracle VM / Downloads.

Oracle Set-up

Before we start the Oracle VM Manager installer we need to consider some perquisites.  OVM requires an oracle user that is a member of a group named dba, iptables-services and ntp installed and a software install directory of /u01, if these have not previously been set-up they can be done with following:

# groupadd dba
# useradd -g dba oracle
# yum install iptables-services
# yum install ntp
# mkdir /u01

Mount OracleVM Manager ISO

We now need to create a mount point  and mount the OVM ISO via loopback.

# mkdir /media/cdrom
# mount -o loop,ro ovmm-3.4.6-installer-OracleLinux-b2105.iso /media/cdrom

Now change directory to where your mounted OracleVM image and run the createOracle.sh script to verify and set Oracle VM Manager configuration parameters, this is based upon the oracle-validated package

[root@z-ovmm cdrom]# ./createOracle.sh 
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.
Adding group 'oinstall' with gid '54321' ...
Adding group 'dba'
Adding user 'oracle' with user id '54321', initial login group 'dba', supplementary group 'oinstall' and  home directory '/home/oracle' ...
Changing ownership of '/home/oracle' to oracle:dba
Creating user 'oracle' succeeded ... 
For security reasons, no default password was set for user 'oracle'. If you wish to login as the 'oracle' user, you will need to set a password for this account.
Verifying user 'oracle' OS prerequisites for Oracle VM Manager ...
oracle soft nofile 8192
oracle hard nofile 65536
oracle soft nproc 2048
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle soft core unlimited
oracle hard core unlimited
Setting  user 'oracle' OS limits for Oracle VM Manager ...
Altered file /etc/security/limits.conf
Original file backed up at /etc/security/limits.conf.orabackup
Verifying & setting of user limits succeeded ... 
Changing '/u01' permission to 755 ...
Modifying iptables for OVM
Adding rules to enable access to:
     7002  : Oracle VM Manager https
       123 : NTP
     10000 : Oracle VM Manager CLI Tool
Redirecting to /bin/systemctl status iptables.service
Redirecting to /bin/systemctl status iptables.service
Redirecting to /bin/systemctl restart iptables.service
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
Redirecting to /bin/systemctl restart iptables.service
Rules added.

Oracle VM Manager Demo Install

As root run the Oracle VM Manager installer, this is character based and does not require a X session unlike other Oracle installers.

[root@z-ovmm cdrom]# ./runInstaller.sh 

Oracle VM Manager Release 3.4.6 Installer

Oracle VM Manager Installer log file:
/var/log/ovmm/ovm-manager-3-install-2018-12-10-141029.log

Please select an installation type:
   1: Install
   2: Upgrade
   3: Uninstall
   4: Help

   Select Number (1-4): 1

Verifying installation prerequisites ...

Starting production with local database installation ...

One password is used for all users created and used during the installation.
Enter a password for all logins used during the installation: 
Enter a password for all logins used during the installation (confirm): 

Please enter your fully qualified domain name, e.g. ovs123.us.oracle.com, (or IP address) of your management server for SSL certification generation, more than one IP address are detected: 10.225.112.213 192.168.122.1 [z-ovmm.uklab.purestorage.com]:  z-ovmm.uklab.purestorage.com

Verifying configuration ...

Start installing Oracle VM Manager:
   1: Continue
   2: Abort

   Select Number (1-2): 1

Step 1 of 7 : Database Software ...
Installing Database Software...
Retrieving MySQL Database 5.6 ...
Unzipping MySQL RPM File ...
Installing MySQL 5.6 RPM package ...
Configuring MySQL Database 5.6 ...
Installing MySQL backup RPM package ...

Step 2 of 7 : Java ...
Installing Java ...

Step 3 of 7 : WebLogic and ADF ...
Retrieving Oracle WebLogic Server 12c and ADF ...
Installing Oracle WebLogic Server 12c and ADF ...
Applying patches to Weblogic ...
Applying patch to ADF ...

Step 4 of 7 : Oracle VM ...
Installing Oracle VM Manager Core ...
Retrieving Oracle VM Manager Application ...
Extracting Oracle VM Manager Application ...

Retrieving Oracle VM Manager Upgrade tool ...
Extracting Oracle VM Manager Upgrade tool ...
Installing Oracle VM Manager Upgrade tool ...

Retrieving Oracle VM Manager CLI tool ...
Extracting Oracle VM Manager CLI tool...
Installing Oracle VM Manager CLI tool ...
Installing Oracle VM Manager WLST Scripts ...

Step 5 of 7 : Domain creation ...
Creating domain ...

Step 6 of 7 : Oracle VM Tools ...

Retrieving Oracle VM Manager Shell & API ...
Extracting Oracle VM Manager Shell & API ...
Installing Oracle VM Manager Shell & API ...

Retrieving Oracle VM Manager Wsh tool ...
Extracting Oracle VM Manager Wsh tool ...
Installing Oracle VM Manager Wsh tool ...

Retrieving Oracle VM Manager Tools ...
Extracting Oracle VM Manager Tools ...
Installing Oracle VM Manager Tools ...

Retrieving ovmcore-console ...
Installing ovmcore-console RPM package ...
Copying Oracle VM Manager shell to '/usr/bin/ovm_shell.sh' ...
Installing ovm_admin.sh in '/u01/app/oracle/ovm-manager-3/bin' ...
Installing ovm_upgrade.sh in '/u01/app/oracle/ovm-manager-3/bin' ...

Step 7 of 7 : Start OVM Manager ...
Enabling Oracle VM Manager service ...
Shutting down Oracle VM Manager instance ...
Starting Oracle VM Manager instance ...

Please wait while WebLogic configures the applications...
Trying to connect to core via ovmwsh (attempt 1 of 20) ...
Trying to connect to core via ovm_shell (attempt 1 of 5)...
Oracle VM Manager installed.

Installation Summary
--------------------
Database configuration:
  Database type               : MySQL
  Database host name          : localhost
  Database name               : ovs
  Database listener port      : 49500
  Database user               : ovs

Weblogic Server configuration:
  Administration username     : weblogic

Oracle VM Manager configuration:
  Username                    : admin
  Core management port        : 54321
  UUID                        : 0004fb0000010000e02391308bcca60b


Passwords:
There are no default passwords for any users. The passwords to use for Oracle VM Manager, Database, and Oracle WebLogic Server have been set by you during this installation. In the case of a default install, all passwords are the same.

Oracle VM Manager UI:
  https://z-ovmm.uklab.purestorage.com:7002/ovm/console
Log in with the user 'admin', and the password you set during the installation.

For more information about Oracle Virtualization, please visit:
  http://www.oracle.com/virtualization/

3.2.10/3.2.11 Oracle VM x86 Servers and SPARC agent 3.3.1 managed Servers are no longer supported in Oracle VM Manager 3.4. Please upgrade your Server to a more current version for full support
For instructions, see the Oracle VM 3.4 Installation and Upgrade guide. 

Oracle VM Manager installation complete.

Please remove configuration file /tmp/ovm_configY49cOf.

You can now login using the URL shown above e.g. https://z-ovmm.uklab.purestorage.com:7002/ovm/console

Oracle VM Manager 
Oracle VM Manager Getting Started

If you have followed the above steps you should now have a working OracleVM Manager installation. In the series of Blog posts on OVM, the next couple of posts will document the required steps to configure iSCSI and NFS storage, before we look at creating Virtual Machines.

ODC Appreciation Day : International conferences and making new friends

Background

Back in 2016 Tim aka ‘ORACLE-BASE’ launched the the first OTN (Oracle Technology Network) Appreciation Day, after the success of 2016 it was repeated again in 2017 under the new moniker of ODC (Oracle Developer Community).

Oracle Developer Community 2018

My ODC 2018 Blog post this year is about the amazing global Oracle community and the various events put on across the planet.

As an OracleAce and speaker, I am fortunate to get the opportunity to visit some fantastic countries, cities and venues. Experience different cultures, try local foods and make new friends, everywhere I go, the Oracle community is warm and welcoming and filled with passionate people who want to share experiences and ideas learn new things.

This year I have created some amazing memories, some of the highlights include:

  • Skydiving at MadridFly
  • A tour of Jerusalem
  • Experiencing a genuinue Finnish sauna and impromptu singing.
  • Drinking Polish vodka on a picture-perfect beach.

And many more, however the best thing about attending international conferences is catching up with old friends and hopefully making new ones

Looking forward to making some new memories and of course new friends at Oracle Open World in a couple of week. #SeeYouAtOpenWorld

On-line migration of Oracle 18c filesystem datafiles to Oracle ASM

Background

Prior to Oracle 12c moving non-ASM data files required taking data files ‘offline’ and negotiating an outage with the business, since Oracle 12c this is has no longer been the case.

I was recently challenged with moving a good size filesystem 18c Oracle database from an existing storage platform to a new storage array.

Online_Data_Move

Prepare ASM Disk Groups

Use sqlplus to create the ASM disk group directories e.g.
$ export ORACLE_SID=+ASM
$ sqlplus / as sysasm

Disk Group +DATA

SQL> ALTER DISKGROUP DATA ADD DIRECTORY ‘+DATA/PSTG’;
SQL> ALTER DISKGROUP DATA ADD DIRECTORY ‘+DATA/PSTG/DATAFILE’;
SQL> ALTER DISKGROUP DATA ADD DIRECTORY ‘+DATA/PSTG/CHANGETRACKING’;
SQL> ALTER DISKGROUP DATA ADD DIRECTORY ‘+DATA/PSTG/TEMPFILE’;

Disk Group +CONTROL_REDO

SQL> ALTER DISKGROUP CONTROL_REDO ADD DIRECTORY ‘+CONTROL_REDO/PSTG’;
SQL> ALTER DISKGROUP CONTROL_REDO ADD DIRECTORY ‘+CONTROL_REDO/PSTG/CONTROLFILE’;
SQL> ALTER DISKGROUP CONTROL_REDO ADD DIRECTORY ‘+CONTROL_REDO/PSTG/ONLINELOG’;
SQL> ALTER DISKGROUP CONTROL_REDO ADD DIRECTORY ‘+CONTROL_REDO/PSTG/PARAMETERFILE’;

Disk Group +FRA

SQL> ALTER DISKGROUP FRA ADD DIRECTORY ‘+FRA/PSTG’;
SQL> ALTER DISKGROUP FRA ADD DIRECTORY ‘+FRA/PSTG/ARCHIVELOG’;

Identify Datafiles

column dname heading ‘Data File Name’ format a45
column filename heading ‘BCT File Name’ format a45
column tablespace_name heading ‘Tablespace Name’ format a25
column data_mb heading ‘Data MB’ format a10

select file_name as dname, tablespace_name, to_char(bytes/1024/1024,’999,999′) as data_mb from dba_data_files

Data File Name                             Tablespace Name Data MB
———————————————-  ————————- ——
/u02/oradata/PSTG/system01.dbf             SYSTEM  880
/u02/oradata/PSTG/sysaux01.dbf             SYSAUX  990
/u02/oradata/PSTG/sh.dbf                                   SH  512,000
/u02/oradata/PSTG/users01.dbf                  USERS  5
/u02/oradata/PSTG/soe.dbf                              SOE  512,000
/u02/oradata/PSTG/undotbs01.dbf    UNDOTBS1  400

On-line Datafile move

I have a previously blogged on using the Oracle 12c ALTER DATABASE MOVE DATAFILE command and you can see the full syntax  on the link above.

SQL> ALTER DATABASE MOVE DATAFILE ‘/u02/oradata/PSTG/system01.dbf’ TO ‘+DATA/PSTG/DATAFILE/system01.dbf’;
SQL> ALTER DATABASE MOVE DATAFILE ‘/u02/oradata/PSTG/sysaux01.dbf’ TO ‘+DATA/PSTG/DATAFILE/sysaux01.dbf’;
SQL> ALTER DATABASE MOVE DATAFILE ‘/u02/oradata/PSTG/sh.dbf’ TO ‘+DATA/PSTG/DATAFILE/sh.dbf’;
SQL> ALTER DATABASE MOVE DATAFILE ‘/u02/oradata/PSTG/users01.dbf’ TO ‘+DATA/PSTG/DATAFILE/users01.dbf’;
SQL> ALTER DATABASE MOVE DATAFILE ‘/u02/oradata/PSTG/undotbs01.dbf’ TO ‘+DATA/PSTG/DATAFILE/undotbs01.dbf’;

Monitor Move Progress

Below is a query which uses the V$SESSION_LONOPS view to report progress of the Online data file move operations.

column st heading ‘Start Time’ format a25
column time_remaining heading ‘Time|Remaining’
column elapsed_seconds heading ‘Elaspsed|Seconds’
column sofar heading ‘Sofar’ format a10
column total heading ‘Total’ format a10
column progress heading ‘Progress’ format a10
column message heading ‘Message’ format a100

select to_char(start_time,’DD-MON-YY hh24:mi:ss’) as st, time_remaining, elapsed_seconds, to_char(SOFAR/1024/1024,’999,999′) || ‘MB’ as sofar, to_char(TOTALWORK/1024/1024,’999,999′) || ‘MB’ as total, to_char((SOFAR/TOTALWORK)*100,’999′) || ‘%’ as progress, message from V$SESSION_LONGOPS;

Example Output

09-AUG-18 14:58:33 0 7 880MB 880MB 100%
Online data file move: data file 1: 922746880 out of 922746880 bytes done

09-AUG-18 15:01:07 0 7 990MB 990MB 100%
Online data file move: data file 3: 1038090240 out of 1038090240 bytes done

09-AUG-18 15:06:34 0 2767 512,000MB 512,000MB 100%
Online data file move: data file 5: 536870912000 out of 536870912000 bytes done

09-AUG-18 15:57:07 2757 38 6,962MB 512,000MB 1%
Online data file move: data file 2: 7300186112 out of 536870912000 bytes done

Updated Data file locations

We can check the file haven been relocated using the same query we ran earlier to identify the datafiles.

select file_name as dname, tablespace_name, to_char(bytes/1024/1024,’999,999′) as data_mb from dba_data_files;

Data File Name Tablespace Name Data MB
——————————————— ————————- ——–
+DATA/PSTG/DATAFILE/system01.dbf SYSTEM 880
+DATA/PSTG/DATAFILE/sysaux01.dbf SYSAUX 1,040
+DATA/PSTG/DATAFILE/sh.dbf SH 512,000
+DATA/PSTG/DATAFILE/users01.dbf USERS 5
+DATA/PSTG/DATAFILE/soe.dbf SOE 512,000
+DATA/PSTG/DATAFILE/undotbs01.dbf UNDOTBS1 400

Creating new Oracle 18c ASM Disk Groups

Creating New Volumes

Create the new ASM volume(s) using your storage platform, for this blog I have created 3 volumes on a Pure FlashArray, simply providing names and sizes.

FA_createVol
Create new ASM Volume(s)

Note the Volume serial numbers,  these will be used in the multipath configuration wwid (World Wide Identifier)

FA_vols
New ASM volumes

SCSI storage device serial numbers are universally unique. Each one starts with a 36bit (9 hexadecimal digit) vendor ID. Because the vendor ID is always the same for Pure Storage FlashArray volumes (3624a9370), it is omitted from the CLI and GUI displays.

Linux Configuration

Create multipath entries in  ‘/etc/multipath.conf’ for the 3 new volumes, in the examples below the wwid (World Wide Identifier) is set to Vendor ID + Serial number. e.g. ‘3624a9370’ + ‘ff551cc46aee54bf9000126ab’.
Note, the ‘wwid’ needs to be in lowercase and the ‘alias’ name for ASM disks needs to be less than 30 characters, alphanumeric and only use the ‘_ ‘ special character. 

multipath

Update multipath Configuration

Flush unused multipath devices

# multipath -F

Reload multipath configuration:

# service multipathd reload

Load and display multipath configuration, device mapper and other components

# multipath -v2
# multipath -ll

Oracle ASM Disk Discovery

Before we update our UDEV rules, lets check our Oracle ASM discovery disk string ‘ASM_DISKSTRING’.

Using asmcmd

ASMCMD> dsget
parameter:/u01/shared_config/ocr_configuration,  /dev/pureasm
profile:/u01/shared_config/ocr_configuration, /dev/pureasm

Using sqlplus

SQL> show parameter ASM_DISKSTRING;

asm_diskstring string /u01/shared_config/ocr_configuration, /dev/pureasm

We can now edit or create a UDEV rules file to reflect the ASM discovery string destination, the SYMLINK prefixes .

udev
/etc/udev/rules.d/99-oracle-asmdevices.rules

Reload UDEV rules as the root user using ‘udevadm trigger’ to set ownership and create links to the ASM disk discovery directory.

asmdisks
ls -la /dev/pureasm

We can now ready to create the ASM Disk Groups using the SQLPlus or the Oracle ASM Configuration Assistant ‘asmca’ UI.

I repeated this 3 times, creating +DATA. +FRA and +CONTROL_REDO which I will use in my next Blog Post, where I plan to share how to migrate from a file systems to Oracle ASM.

asmca
asmca