—————– Installing Oracle RAC release 11.2 under CentOS —————-
This tutorial was tested under CentOS x86_64 kernel 2.6.18-194.3.1.el5
this version is old but it matches the ASMlib .
also this tutorial was deploy on one node only , if there are more nodes
in the cluster , a shared storage should be used . oracle support NAS,SAN,NFS,RAW DEVICES etc.
Table of content
1. Preparing OS properties
2. Creating users and groups
3. Installing and setting ASMLib
4. verifying Grid Infrastructure preparation
5. Installing Grid Infrastructure
6. Installing Database software
7. creating database
1. Preparing the operating system parameters
list of packages require to be installed on all nodes
the versions should be >=
binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
gcc-4.1.1
gcc-c++-4.1.1
glibc-2.5-12
glibc-2.5-12 (32 bit)
glibc-common-2.5
glibc-devel-2.5
glibc-devel-2.5-12 (32 bit)
libaio-0.3.106
libaio-0.3.106 (32 bit)
libaio-devel-0.3.106
libgcc-4.1.1
libgcc-4.1.1 (32 bit)
libstdc++-4.1.1
libstdc++-4.1.1 (32 bit)
libstdc++-devel 4.1.1
make-3.81
sysstat-7.0.0
yum command :
[root]# yum install compat-libstdc++-33.i386 compat-libstdc++-33.x86_64 elfutils-libelf.x86_64 elfutils-libelf-devel.x86_64 glibc.i686 glibc.x86_64 glibc-common.x86_64 glibc-devel.i386 glibc-devel.x86_64 libaio.i386 libaio.x86_64 libaio-devel.x86_64 libgcc.i386 libgcc.x86_64 libstdc++.i386 libstdc++.x86_64 libstdc++-devel.x86_64 binutils gcc gcc-c++ make -y http://ftp.pbone.net/mirror/ftp.owlriver.com/pub/local/ORC/dummy-redhat-release/dummy-redhat-release-1-1orc.src.rpm
1.1 First thing is to set all recommended kernel parameters
edit the file /etc/sysctl.conf , and change/add this parameters
# Oracle Setting kernel.sem=250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65000 net.core.rmem_default = 4194304 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 fs.file-max = 6815744 fs.aio-max-nr=1048576
Then run the command sysctl -p as root
so that the new settings will be load into the kernel parameters
1.2 Setting shared partition permissions , in order for ASM/Oracle to gain ownership
In this step it depend on the partitions you have attached from the storage .
here we are using local partitions but its the same , we gonna use 3 partitions
sda6,7,8 where sda6 will be for voting and OCR as ASM , sda7 will be for undo and sda8 will hold the data
[root@~]# fdisk -lDisk /dev/sda: 896.9 GB, 896998047744 bytes 255 heads, 63 sectors/track, 109053 cylinders Units = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System /dev/sda1 * 1 2550 20482843+ 83 Linux /dev/sda2 2551 5100 20482875 83 Linux /dev/sda3 5101 6120 8193150 82 Linux swap / Solaris /dev/sda4 6121 109053 826809322+ 5 Extended /dev/sda5 6121 18279 97667136 83 Linux /dev/sda6 18280 18402 987966 83 Linux /dev/sda7 18403 18525 987966 83 Linux /dev/sda8 18526 30684 97667136 83 Linux
Create a udev rule for ownership and permissions , by creating a file /etc/udev/rules.d/51-oracle.permissions.rules
and entering this lines :
# OCR disks KERNEL=="sda6" , OWNER="grid" GROUP="asmdba" , MODE="0660" # UNDO disks KERNEL=="sda7" , OWNER="grid" GROUP="asmdba" , MODE="0660" # DATA disks KERNEL=="sda8" , OWNER="grid" GROUP="asmdba" , MODE="0660"
after creating the file , the system must reboot (but first create grid user in step 2), after reboot the device should look like this :
[[email protected] app]# ls -l /dev/sda* brw-r----- 1 root disk 8, 0 Mar 30 16:17 /dev/sda brw-r----- 1 root disk 8, 1 Mar 30 16:17 /dev/sda1 brw-r----- 1 root disk 8, 2 Mar 30 16:17 /dev/sda2 brw-r----- 1 root disk 8, 3 Mar 30 16:17 /dev/sda3 brw-r----- 1 root disk 8, 4 Mar 30 16:17 /dev/sda4 brw-r----- 1 root disk 8, 5 Mar 30 16:17 /dev/sda5 brw-rw---- 1 grid asmdba 8, 6 Mar 30 16:17 /dev/sda6 brw-rw---- 1 grid asmdba 8, 7 Mar 30 16:17 /dev/sda7 brw-rw---- 1 grid asmdba 8, 8 Mar 30 16:17 /dev/sda8
1.2.1 Network settings ,
each node must have at least 2 interfaces , one for public and one for interconnect
also each node must have 3 hosts resolved for example
eth0 = 172.16.2.14/24 (public)
eth0:0 = 172.16.2.15/24 (vip) no need to set , but needs to be resolved
eth1 = 10.11.12.1/24 (private)
etc/hosts should look like this :
172.16.2.14 dbtest.example.com dbtest
172.16.2.15 dbtest-vip.example.com dbtest-vip
10.11.12.1 dbtest-priv.example.com dbtest-priv
1.3 DNS settings ,
for Oracle version 11.2 we need to set a SCAN ip and VIP in the DNS,
Oracle recommend that we set 3 ip for SCAN
SCAN ip should be part of the host domain for example
dbtest.example.com IN A 172.16.2.14
dbtest-priv.example.com IN A 10.11.12.1
dbtest-vip.example.com IN A 172.16.2.15
scan.dbtest.example.com IN A 172.16.2.17
scan.dbtest.example.com IN A 172.16.2.18
scan.dbtest.example.com IN A 172.16.2.19
* its a good idea to set all the domains in DNS ,
* If you want to use GNS you need to have DHCP as well ( we are not going to use GNS here )
1.4 fake CentOS as Redhat
Setting OS as fake REDHAT , backup this files :
/etc/issue and /etc/redhat-release
edit this files and replace the CentOS with this line
Red Hat Enterprise Linux Server release 5 (Tikanga)
also install the dummy package redhat-release dummy
can be found here as spec file and compile it,
or you can download the RPM i had compile from that spec from this site here
and last thing is running this command , for the grid install process
[root@~]# echo "redhat-release-5Server-5" > /tmp/.linux_release [root@~]# chattr +i /tmp/.linux_release
2. Creating users and groups ,
we need to create 2 users one for oracle grid system
and one for the database , lets call this users grid and oracle for simplicity .
first create the users , then create the groups :
groupadd oinstall groupadd dba groupadd asmadmin groupadd asmdba groupadd asmoper groupadd oper useradd -g oinstall -G dba,oper,asmdba oracle useradd -g oinstall -G dba,asmoper,asmdba,asmadmin grid
2.1 setting ssh connections between all nodes with no password
all nodes must be free access via ssh from the install server , for that we need to create ssh keys
for both grid and oracle user , by running ssh-keygen and copying the certificates between all nodes
as append to the file ~/.ssh/authorized_keys
2.2 Setting up limits for oracle and grid users .
by default CentOS have a open file limit of 1024 and process ,
this limit will not do for oracle .
setting limits is done by editing the file /etc/security/limits.conf .
oracle and grid users should have unlimited processes and open files
grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536
3. Download and install ASMlib can be found at
Oracle.com
select the corrent kernel version
in our case
oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
oracleasm-support-2.1.3-1.el5.x86_64.rpm
3.1 after installing ASM we need to configure and create our ASM partitions ,
make sure you choose the correct user and group for ASM
[root@~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.Default user to own the driver interface [grid]:
Default group to own the driver interface [asmdba]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ][[email protected] app]# /etc/init.d/oracleasm listdisks
[[email protected] app]# /etc/init.d/oracleasm createdisk OCR /dev/sda6
Marking disk "OCR" as an ASM disk: [ OK ]
[[email protected] app]# /etc/init.d/oracleasm createdisk UNDO /dev/sda7
Marking disk "UNDO" as an ASM disk: [ OK ]
[[email protected] app]# /etc/init.d/oracleasm createdisk DATA /dev/sda8
Marking disk "DATA" as an ASM disk: [ OK ]
[[email protected] app]# /etc/init.d/oracleasm listdisks
DATA
OCR
UNDO[[email protected] app]# ls -l /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmdba 8, 8 Mar 31 10:33 DATA
brw-rw---- 1 grid asmdba 8, 6 Mar 31 10:33 OCR
brw-rw---- 1 grid asmdba 8, 7 Mar 31 10:33 UNDO
4. Unzip the grid infrastructure under grid user home directory
cd into grid folder and run the runcluvfy.sh with parameters
./runcluvfy.sh stage -pre crsinst -n <node name>
you can also create a fixup file that can help on solving all problems
4.1 solving NTP issue
by edit the file /etc/sysconfig/ntpd
and adding -x to ntp options
OPTIONS=”-x -u ntp:ntp -p /var/run/ntpd.pid”
5. run the Grid Infrastructure installer and follow this guide lines
create 2 directories , one for Grid and one for the Software , and set permissions
note that the grid and software has different homes !
[root@~]# mkdir -p /opt/app/OCR/base [root@~]# mkdir -p /opt/app/OCR/software [root@~]# chown -R grid:oinstall /opt/app/OCR[root@~]# mkdir -p /opt/app/DB [root@~]# chown -R oracle:oinstall /opt/app/DB
5.1 next steps are part of the Grid installer
a. choose “Install and configure Grid Infrastructure for cluster”
b. choose the “Advance installation”
c. choose English and next
d. set this settings :
Cluster name : dbtest_cluster
SCAN name : scan.dbtest.example.com
SCAN port : 1521
* uncheck use of GNS
e. select the nodes you want to install
f. make sure the correct interfaces are set with public and private
g. choose the OCR to be installed on ASM
h. now we choose the ASM partition we dedicated for OCR and name the diskgroup OCR
i. set both password the same for SYS and ASMSNMP
j. do not install IPMI
k. set the os groups for management
OSDBA= asmdba
OSOPER= asmoper
OSASM=asmadmin
l. set oracle base ( /opt/app/OCR/base ) , and software base ( /opt/app/OCR/software ) from stage 5.
m. set the Inventory folder ( i choose the default ) /opt/app/oraInventory
o. the check may fail on some parameters , fix and continue
p. you will be prompted for running root.sh from root terminal , all should come clean ,
and must be run on all nodes before you could continue .
before running the root.sh script , you must edit the file $OCR_HOME/lib/osds_acfslib.pm
and change this lines to support CentOS , or you will get an error
ADVM/ACFS is not supported on centos-release-5-5.el5.centos
so edit the file and make sure CentOS is one of the supported OS
if (($release =~ /^redhat-release/) || # straight RH ($release =~ /^enterprise-release/) || # Oracle Enterprise Linux ($release =~ /^centos-release/)) # CentOS
/opt/app/oraInventory/orainstRoot.sh
/opt/app/OCR/root.sh
q. the install should now finish with no errors !
6. installing database , for that we need to it as oracle user ( very important )
this are the steps from the installer GUI
a. install the database software only , later we will create a database
b. choose the RAC and the nodes involved . and test ssh conectivity
c. choose English and continue
d. we use Standard edition
e. here we set the ORACLE_BASE as we did in the step 5 and ORACLE_HOME under ORACLE_BASE
in our case i choose
ORACLE_BASE = /opt/app/DB
ORACLE_HOME = /opt/app/DB/11.2.0
f. choosing OSDBA and OSOPER as dba,oper
g. fix any warning and continue (or ignore)
h. during install you will be request to run root.sh script as root user
/opt/app/DB/11.2.0/root.sh
7. creating ASM disk groups
by running as user grid $OCR_HOME/bin/asmca
we will create the +UNDO and +DATA
8. Create a database by
running the $ORACLE_HOME/bin/dbca as user oracle