Installing Oracle RAC 11.2 under CentOS

—————– Installing Oracle RAC release 11.2 under CentOS —————-
This tutorial was tested under CentOS x86_64 kernel 2.6.18-194.3.1.el5
this version is old but it matches the ASMlib .
also this tutorial was deploy on one node only , if there are more nodes
in the cluster , a shared storage should be used . oracle support NAS,SAN,NFS,RAW DEVICES etc.
Table of content
1. Preparing OS properties
2. Creating users and groups
3. Installing and setting ASMLib
4. verifying Grid Infrastructure preparation
5. Installing Grid Infrastructure
6. Installing Database software
7. creating database
1. Preparing the operating system parameters
list of packages require to be installed on all nodes
the versions should be >=
compat-libstdc++-33-3.2.3 (32 bit)
glibc-2.5-12 (32 bit)
glibc-devel-2.5-12 (32 bit)
libaio-0.3.106 (32 bit)
libgcc-4.1.1 (32 bit)
libstdc++-4.1.1 (32 bit)
libstdc++-devel 4.1.1
yum command :

[root]# yum install compat-libstdc++-33.i386 compat-libstdc++-33.x86_64 elfutils-libelf.x86_64 elfutils-libelf-devel.x86_64 glibc.i686 glibc.x86_64 glibc-common.x86_64 glibc-devel.i386 glibc-devel.x86_64 libaio.i386 libaio.x86_64 libaio-devel.x86_64 libgcc.i386 libgcc.x86_64 libstdc++.i386 libstdc++.x86_64 libstdc++-devel.x86_64 binutils gcc gcc-c++ make -y

1.1 First thing is to set all recommended kernel parameters
edit the file /etc/sysctl.conf , and change/add this parameters

# Oracle Setting
kernel.sem=250        32000   100      128
net.ipv4.ip_local_port_range = 9000 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.file-max = 6815744

Then run the command sysctl -p as root
so that the new settings will be load into the kernel parameters
1.2 Setting shared partition permissions , in order for ASM/Oracle to gain ownership
In this step it depend on the partitions you have attached from the storage .
here we are using local partitions but its the same , we gonna use 3 partitions
sda6,7,8 where sda6 will be for voting and OCR as ASM , sda7 will be for undo and sda8 will hold the data

[root@~]# fdisk -lDisk /dev/sda: 896.9 GB, 896998047744 bytes
255 heads, 63 sectors/track, 109053 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytesDevice Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2550    20482843+  83  Linux
/dev/sda2            2551        5100    20482875   83  Linux
/dev/sda3            5101        6120     8193150   82  Linux swap / Solaris
/dev/sda4            6121      109053   826809322+   5  Extended
/dev/sda5            6121       18279    97667136   83  Linux
/dev/sda6           18280       18402      987966   83  Linux
/dev/sda7           18403       18525      987966   83  Linux
/dev/sda8           18526       30684    97667136   83  Linux

Create a udev rule for ownership and permissions , by creating a file /etc/udev/rules.d/51-oracle.permissions.rules
and entering this lines :

# OCR disks
KERNEL=="sda6" , OWNER="grid" GROUP="asmdba" , MODE="0660"
# UNDO disks
KERNEL=="sda7" , OWNER="grid" GROUP="asmdba" , MODE="0660"
# DATA disks
KERNEL=="sda8" , OWNER="grid" GROUP="asmdba" , MODE="0660"

after creating the file , the system must reboot (but first create grid user in step 2), after reboot the device should look like this :

[[email protected] app]# ls -l /dev/sda*
brw-r----- 1 root disk     8, 0 Mar 30 16:17 /dev/sda
brw-r----- 1 root disk     8, 1 Mar 30 16:17 /dev/sda1
brw-r----- 1 root disk     8, 2 Mar 30 16:17 /dev/sda2
brw-r----- 1 root disk     8, 3 Mar 30 16:17 /dev/sda3
brw-r----- 1 root disk     8, 4 Mar 30 16:17 /dev/sda4
brw-r----- 1 root disk     8, 5 Mar 30 16:17 /dev/sda5
brw-rw---- 1 grid asmdba 8, 6 Mar 30 16:17 /dev/sda6
brw-rw---- 1 grid asmdba 8, 7 Mar 30 16:17 /dev/sda7
brw-rw---- 1 grid asmdba 8, 8 Mar 30 16:17 /dev/sda8

1.2.1 Network settings ,
each node must have at least 2 interfaces , one for public and one for interconnect
also each node must have 3 hosts resolved for example
eth0 = (public)
eth0:0 = (vip) no need to set , but needs to be resolved
eth1 = (private)
etc/hosts should look like this : dbtest dbtest-vip dbtest-priv
1.3 DNS settings ,
for Oracle version 11.2 we need to set a SCAN ip and VIP in the DNS,
Oracle recommend that we set 3 ip for SCAN
SCAN ip should be part of the host domain for example IN A IN A IN A IN A IN A IN A
* its a good idea to set all the domains in DNS ,
* If you want to use GNS you need to have DHCP as well ( we are not going to use GNS here )
1.4 fake CentOS as Redhat
Setting OS as fake REDHAT , backup this files :
/etc/issue and /etc/redhat-release
edit this files and replace the CentOS with this line
Red Hat Enterprise Linux Server release 5 (Tikanga)
also install the dummy package redhat-release dummy
can be found here as spec file and compile it,
or you can download the RPM i had compile from that spec from this site here
and last thing is running this command , for the grid install process

[root@~]# echo "redhat-release-5Server-5" > /tmp/.linux_release
[root@~]# chattr +i /tmp/.linux_release

2. Creating users and groups ,
we need to create 2 users one for oracle grid system
and one for the database , lets call this users grid and oracle for simplicity .
first create the users , then create the groups :

groupadd oinstall
groupadd dba
groupadd asmadmin
groupadd asmdba
groupadd asmoper
groupadd oper
useradd -g oinstall -G dba,oper,asmdba oracle
useradd -g oinstall -G dba,asmoper,asmdba,asmadmin grid

2.1 setting ssh connections between all nodes with no password
all nodes must be free access via ssh from the install server , for that we need to create ssh keys
for both grid and oracle user , by running ssh-keygen and copying the certificates between all nodes
as append to the file ~/.ssh/authorized_keys
2.2 Setting up limits for oracle and grid users .
by default CentOS have a open file limit of 1024 and process ,
this limit will not do for oracle .
setting limits is done by editing the file /etc/security/limits.conf .
oracle and grid users should have unlimited processes and open files

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

3. Download and install ASMlib can be found at
select the corrent kernel version
in our case
3.1 after installing ASM we need to configure and create our ASM partitions ,
make sure you choose the correct user and group for ASM

[root@~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface [grid]:
Default group to own the driver interface [asmdba]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ][[email protected] app]# /etc/init.d/oracleasm listdisks
[[email protected] app]# /etc/init.d/oracleasm createdisk OCR /dev/sda6
Marking disk "OCR" as an ASM disk:                         [  OK  ]
[[email protected] app]# /etc/init.d/oracleasm createdisk UNDO /dev/sda7
Marking disk "UNDO" as an ASM disk:                        [  OK  ]
[[email protected] app]# /etc/init.d/oracleasm createdisk DATA /dev/sda8
Marking disk "DATA" as an ASM disk:                        [  OK  ]
[[email protected] app]# /etc/init.d/oracleasm listdisks
UNDO[[email protected] app]# ls -l /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmdba 8, 8 Mar 31 10:33 DATA
brw-rw---- 1 grid asmdba 8, 6 Mar 31 10:33 OCR
brw-rw---- 1 grid asmdba 8, 7 Mar 31 10:33 UNDO

4. Unzip the grid infrastructure under grid user home directory
cd into grid folder and run the with parameters
./ stage -pre crsinst -n <node name>
you can also create a fixup file that can help on solving all problems
4.1 solving NTP issue
by edit the file /etc/sysconfig/ntpd
and adding -x to ntp options
OPTIONS=”-x -u ntp:ntp -p /var/run/”
5. run the Grid Infrastructure installer and follow this guide lines
create 2 directories , one for Grid and one for the Software , and set permissions
note that the grid and software has different homes !

[root@~]# mkdir -p /opt/app/OCR/base
[root@~]# mkdir -p /opt/app/OCR/software
[root@~]# chown -R grid:oinstall /opt/app/OCR[root@~]# mkdir -p /opt/app/DB
[root@~]# chown -R oracle:oinstall /opt/app/DB

5.1 next steps are part of the Grid installer
a. choose “Install and configure Grid Infrastructure for cluster”
b. choose the “Advance installation”
c. choose English and next
d. set this settings :
Cluster name : dbtest_cluster
SCAN name :
SCAN port : 1521
* uncheck use of GNS
e. select the nodes you want to install
f. make sure the correct interfaces are set with public and private
g. choose the OCR to be installed on ASM
h. now we choose the ASM partition we dedicated for OCR and name the diskgroup OCR
i. set both password the same for SYS and ASMSNMP
j. do not install IPMI
k. set the os groups for management
OSDBA= asmdba
OSOPER= asmoper
l. set oracle base ( /opt/app/OCR/base ) , and software base ( /opt/app/OCR/software ) from stage 5.
m. set the Inventory folder ( i choose the default ) /opt/app/oraInventory
o. the check may fail on some parameters , fix and continue
p. you will be prompted for running from root terminal , all should come clean ,
and must be run on all nodes before you could continue .
before running the script , you must edit the file $OCR_HOME/lib/
and change this lines to support CentOS , or you will get an error

ADVM/ACFS is not supported on centos-release-5-5.el5.centos

so edit the file and make sure CentOS is one of the supported OS

if (($release =~ /^redhat-release/) || # straight RH
($release =~ /^enterprise-release/) || # Oracle Enterprise Linux
($release =~ /^centos-release/)) # CentOS

q. the install should now finish with no errors !
6. installing database , for that we need to it as oracle user ( very important )
this are the steps from the installer GUI
a. install the database software only , later we will create a database
b. choose the RAC and the nodes involved . and test ssh conectivity
c. choose English and continue
d. we use Standard edition
e. here we set the ORACLE_BASE as we did in the step 5 and ORACLE_HOME under ORACLE_BASE
in our case i choose
ORACLE_BASE = /opt/app/DB
ORACLE_HOME = /opt/app/DB/11.2.0
f. choosing OSDBA and OSOPER as dba,oper
g. fix any warning and continue (or ignore)
h. during install you will be request to run script as root user
7. creating ASM disk groups
by running as user grid $OCR_HOME/bin/asmca
we will create the +UNDO and +DATA
8. Create a database by
running the $ORACLE_HOME/bin/dbca as user oracle