oracle-rac安装

http://blog.chinaunix.net/xmlrpc.php?r=blog/article&id=4681351&uid=29655480

参考

1.百度文库中的收藏

2.51cto的视频课程

rac课程学习笔记

安装条件
至少7个ip地址
oel linux AS 6.x x86_64
oracle 11.2.0.4
pl/sql工具

实例与数据库两部分组成

数据库是磁盘上存储的物理文件的集合
实例是预分配的内存与后台进程的集合

数据库包括：控制文件，日志文件，数据文件
还有参数文件，密码文件，归档日志文件。
实例包括：SGA和后台进程。

db_name,service_name,sid,instance_name,db_unique_name的区别
www.itpux.com/thread-236-1-1.html

oracle real application cluster (oracle RAC真正应用集群技术)
在oracle rac环境下，oracle集成了集群软件与存储管理软件。集群软件CRS,存储软件ASM，多个节点共享一份数据。11g叫grid

12c r1 rac重点变化
多了个flex cluster,flex asm的概念，rac的db实例与asm实例不用一一对应了

oracle 11g r2引入了scan,它是一个域名，可以解析至少一个ip,最多3个scan ip.客户端可以通过这个scan 名字来访问数据库。
另外scan ip 必须与public ip和VIP在同一个子网
scan vip：即scan ip,是由dns或者gns,hosts解析出来的ip地址。目前scan vip最多能有三个，它们循环地被客户端所请求到。
gns vip：同scan ip,如果不使用gns解析方法，那么也不会存在gns vip.
scan listener:scan监听器，实现scan的负载均衡功能。如果rac上有三个scan vip,那么scan监听器也有三个，它们各自监听scan
vip的连接请求。

配置scan有三种方法
/etc/hosts，简单易用，目前使用最多的方式。缺点只能对应一个scan ip
dns
gns,一般不常用

scan工作流
在启用scan特性之后，除了多一个或几个scan ip之外，还会多一个或几个scan ip listener,scan ip 对应一个scan ip listener.
客户端连接的时候，会随机的连接到某一个scan ip 及其对应的scan ip listner上，
简单点就是：client>scan listener>locallistener>local instance

存储使用规划
1.grid集群组件磁盘组
+dggrid1:1个1g
+dggrid2:1个1g ocr镜像
2.数据库安装磁盘组
+dgsystem: 用于数据库基本表空间，控制文件，参数文件等
+dgrecover: 用于归档与闪回日志空间
+dgdata01: 用于数据库业务表空间
+dgdata02-x: 用于数据库业务表空间

其实在esxi5.1中不需要修改vmx配置文件，在虚拟机》编辑设置里
将共享磁盘的总线即scsi1的控制器类型改为LSI logic 并行,
scsi总线共享改为物理就行了

下面是不需要的，费了好大劲。因为视频中用的是vmware server虚拟机，而我的环境是esxi5.1,所以这就是差别
disk.locking="FALSE"
diskLib.dataCacheMaxSize="0"
diskLib.dataCacheMaxReadAheadSize="0"
diskLib.dataCacheMinReadAheadSize="0"
diskLib.dataCachePageSize="4096"
diskLib.maxUnsyncedWrites="0"
disk.EnableUUID="TRUE"
scsi1.sharedBus="VIRTUAL"
scsi1.virtualDev="lsilogic"

删除重复的行，用上面的

scsi1.present = "TRUE"
scsi1.sharedBus = "none"
scsi1.virtualDev = "lsilogic"
scsi1:0.present = "TRUE"
scsi1:0.fileName = "/vmfs/volumes/f4cd1698-c50cb6be/131-rac1-cs65/131-rac1-cs65.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.deviceType = "scsi-hardDisk"
sched.scsi1:0.shares = "normal"
sched.scsi1:0.throughputCap = "off"
scsi1:1.present = "TRUE"
scsi1:1.fileName = "/vmfs/volumes/f4cd1698-c50cb6be/131-rac1-cs65/131-rac1-cs65_1.vmdk"
scsi1:1.mode = "independent-persistent"
scsi1:1.deviceType = "scsi-hardDisk"
scsi1:2.present = "TRUE"
scsi1:2.fileName = "/vmfs/volumes/f4cd1698-c50cb6be/131-rac1-cs65/131-rac1-cs65_2.vmdk"
scsi1:2.mode = "independent-persistent"
scsi1:2.deviceType = "scsi-hardDisk"
scsi1:3.present = "TRUE"
scsi1:3.fileName = "/vmfs/volumes/f4cd1698-c50cb6be/131-rac1-cs65/131-rac1-cs65_3.vmdk"
scsi1:3.mode = "independent-persistent"
scsi1:3.deviceType = "scsi-hardDisk"
scsi1:4.present = "TRUE"
scsi1:4.fileName = "/vmfs/volumes/f4cd1698-c50cb6be/131-rac1-cs65/131-rac1-cs65_4.vmdk"
scsi1:4.mode = "independent-persistent"
scsi1:4.deviceType = "scsi-hardDisk"
scsi1:5.present = "TRUE"
scsi1:5.fileName = "/vmfs/volumes/f4cd1698-c50cb6be/131-rac1-cs65/131-rac1-cs65_5.vmdk"
scsi1:5.mode = "independent-persistent"
scsi1:5.deviceType = "scsi-hardDisk"

6课ssh信任与asm磁盘创建

oracle,grid两台节点的ssh互信

su - oracle
rm -rf .ssh
ssh-keygen -t rsa
ssh-keygen -t dsa
cat .ssh/*.pub > .ssh/authorized_keys


a机用户
scp .ssh/authorized_keys 192.168.30.41:~/.ssh/keys_dbs
b机用户
cat .ssh/keys_dbs >> .ssh/authorized_keys
scp .ssh/authorized_keys 192.168.30.31:~/.ssh/

7课安装grid
先安装cvqdisk
再解压grid的安装包
root用户下安装cvuqdisk
export CVUQDISK_GRP=oinstall
rpm -ivh cvuqdisk-1.0.9-1.rpm

报了几个错：shmall与cluster verification utility failed,都跳过了

8课asm磁盘组创建与crs磁盘组的镜像

asmca来创建
新建一个dg,选一个asm磁盘，做成external,然后在两个节点上挂载（右键）。mounted (2of2)就正常了，1of1不对
au为4M

ocrcheck
默认只有dggrid1,
要在root下操作，将dggrid2加入
[grid@rac02 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 7bf2c81459334ffebf01c410d096a047 (ORCL:CRSVOL1) [CRS]
Located 1 voting disk(s).
[grid@rac02 ~]$ ocrconfig -add +dggrid2
只需要在一台主机上做就可以了，这是一块仲裁盘。对这块盘也做了一个镜像。
ocrcheck检查一下，有两块盘了。
因为下面没有做，所以只有一块+CRS
[grid@rac02 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2808
Available space (kbytes) : 259312
ID : 1475542534
Device/File Name : +CRS
Device/File integrity check succeeded

9课安装oracle

oracle用户登录
先解压
./ouInstaller
1.只安装数据库软件，数据库用后面的dbca来安装。
2.默认安装到两个节点上。可以用ssh connectivity来验证一下。

忽略scan 警告，下一步。

10课集群数据库安装
dbca配置界面选择oracle rac database,与单机版的选项是不一样的
自定义数据库
两个名字一般来说是一样的
global database name:itpux
sid prefix:itpux
两个节点

6of12时也就是asm选择时，指定+dgsystem作为软件安装位置。

11课集群日常管理与维护

主要管理工具是srvctl和crsctl这两个工具
srvctl用于管理定义在ocr上的资源
crsctl用于管理群集守护进程，包括cssd,crsd,evmd

crsctl status resource
crs_stat -t
This command is deprecated and has been replaced by 'crsctl status resource'
This command remains for backward compatibility only
crs_stat -p

srvctl status 对象
ocrcheck
ocrconfig -showbackup
ocrconfig -manualbackup
ocrdump -backupfile /u01/app/11.2.0/grid/cdata/rac-cluster/day.ocr
各种群集服务的日志路径
/u01/app/11.2.0/grid/log/rac01

rac启停节点，群集
按顺序操作启动与停止：
停止：关闭数据库，关闭crs

启用crs系统重启而自动启动
crsctl enable crs

crsctl config crs

asm扩展表空间,只需要指定磁盘组名即可

alter tablespace scdata01 add datafile '+dgdata01' size 1M autoextend off;

12课
客户端安装与pl/sql安装
pl/sql需要32位的客户端软件

负载均衡与故障切换测试
停掉其中一台crsctl stop crs
crsctl start crs
crs_stat -t
tnsping itpux1

failover_mode=
type=session
type=select
二者是不一样的
如果切换成功，证明rac就完成了

13课
配置启动归档及闪回

先配置归档才可以启动闪回

先停掉一个数据库，在另外一个上操作，改成非群集模式，
alter system set cluster_database=false scope=spfile

alter system set db_recovery_file_dest_size=2g scope=both
alter system set db_recovery_file_dest='+DGRECOVERY' scope=both
shutdown immediate
startup mount
alter database archivelog
alter database noarchivelog
alter system set cluster_database=true scpoe=spfile

启动闪回
alter database flashback ON;
show parameter recovery

alter database open
alter system switch logfile;

archive log list;

生产环境尽量不开闪回

添加一个卷的方法

存储端添加同一个卷到不同的主机
主机端只需要
iscsiadm -m node -R
parted /dev/mapper/mpathc
mklabel gpt
mkpart primary 0% 100%
p
q

再做一个asm盘
/etc/init.d/oracleasm createdisk rac /dev/mapper/mpathcp1
oracleasm scandisks
oracleasm listdisks

在另一台主机上只需要iscsiadm扫描下,发现盘后，再oracleasm扫描一下,asm盘就出来了



增加其它的asm磁盘组

在rac01上用grid用户运行asmca来创建其它的asm磁盘组，
创建，高级，au是4M，与第一个磁盘组保持一致。
asm平时维护就是看容量，不够时就要扩容了
asmcmd lsdg查看


做镜像

ocrcheck
root用户执行ocrconfig
ocrconfig  -add  +arch    对某一个磁盘组做镜像。

oracle clusterware files包括

ocr和votedisk

OCR is a file (oracle cluster registry)

常用下面这三条命令，grid用户执行

ocrcheck

ocrconfig --showbackup

crsctl query css votedisk

Oracle每四个小时自动发起备份,并保存三个版本,但只存在一个节点上。

关于Voting Disk和OCR Disk 是否需要经常备份的问题。OCR和Votting Disk在安装配置完RAC数据库后就应该进行备份。OCR因该在OCR配置变动之后就备份，比如：变动资源的auto_start设置、添加新的Service等。Voting Disk应该在RAC成员发生变化后备份，比如：增加节点、删除节点。用dd命令做备份。每4小时：保留最后3份；每天结束时：保留最后2份；每周结束时：保留最后2份。OCR要有Mirror，Voting Disk最少做3份，HA考虑。

rac的两个节点的asm信息要一致

[root@rac02 ~]# oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE="sd"
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
[root@rac02 ~]# cat /etc/sysconfig/oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver.  It is generated
# By running /etc/init.d/oracleasm configure.  Please use that method
# to modify this file
#

# ORACLEASM_ENABLED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=asmadmin

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=""

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

udev
重启机器才能生效

vi /etc/udev/rules.d/99-asm.rules
KERNEL=="dm-[3-4]",OWNER="grid",GROUP="asmadmin",MODE="0660"

scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathb

KERNEL=="dm-3", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathb", RESULT=="36005076300810df6f800000000000005", NAME="asm-disk1",  OWNER="grid", GROUP="asmadmin", MODE="0660"


ENV{DM_NAME}=="crs01", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"

dmsetup ls

prvf-5508
prvg-10122



grid用户安装
在3of7中的rpm目录中有cvuqdisk.rpm
[grid@rac01 rpm]$ ls
cvuqdisk-1.0.9-1.rpm
[grid@rac01 rpm]$ pwd
/home/grid/grid/rpm

export CVUQDISK_GRP=oinstall
rpm -ivh /home/grid/grid/rpm/cvuqdisk-1.0.9-1.rpm

不用手动配置racvip，安装grid自动配置

先验证一下环境有无问题
Performing pre-checks for cluster services setup
./runcluvfy.sh stage -pre crsinst -n rac01,rac02 -fixup -verbose
使用cvu 验证硬件和操作系统设置
Performing post-checks for hardware and operating system setup
./runcluvfy.sh stage -post hwos -n rac01,rac02 -verbose

失败项有
1、
Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status                  
  ------------------------------------  ------------------------
  rac02                                 passed                  
  rac01                                 failed                  
Result: PRVF-4007 : User equivalence check failed for user "grid"
2、
Check: TCP connectivity of subnet "10.10.10.0"
  Source                          Destination                     Connected?      
  ------------------------------  ------------------------------  ----------------
  rac01:192.168.30.31             rac02:10.10.10.41               failed          

ERROR: 
PRVF-7617 : Node connectivity between "rac01 : 192.168.30.31" and "rac02 : 10.10.10.41" failed
Result: TCP connectivity check failed for subnet "10.10.10.0"

WARNING: 
Could not find a suitable set of interfaces for the private interconnect

3、
Check: Kernel parameter for "shmmax" 
  Node Name         Current       Configured    Required      Status        Comment     
  ----------------  ------------  ------------  ------------  ------------  ------------
  rac02             536870912     536870912     4294967295    failed        Current value incorrect. Configured value incorrect. 
Result: Kernel parameter check failed for "shmmax"

4、
rac02gcc未装

5
Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
No NTP Daemons or Services were found to be running
PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):
rac02
Result: Clock synchronization check using Network Time Protocol(NTP) failed

rac　grid的安装，只在第一个节点上做，第二个节点只执行两个脚本

第一个节点相关输出

[root@rac01 ntp]# /oracle/app/oraInventory/orainstRoot.sh
Changing permissions of /oracle/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac01 ntp]# /oracle/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /oracle/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac01'
CRS-2676: Start of 'ora.mdnsd' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac01'
CRS-2676: Start of 'ora.gpnpd' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac01'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac01'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac01'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac01'
CRS-2676: Start of 'ora.diskmon' on 'rac01' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac01' succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 85199c8653d34fecbf7b5c20620eaa35.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   85199c8653d34fecbf7b5c20620eaa35 (ORCL:RACALL) [DATA]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac01'
CRS-2676: Start of 'ora.asm' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac01'
CRS-2676: Start of 'ora.DATA.dg' on 'rac01' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded





第二个节点只需要执行以下两个脚本就可以了

[root@rac02 ~]# /oracle/app/oraInventory/orainstRoot.sh
Changing permissions of /oracle/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle/app/oraInventory to oinstall.
The execution of the script is complete.






[root@rac02 ~]# /oracle/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /oracle/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac01, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded



然后在第一个节点上查看如下

[grid@rac01 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac01       
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac01       
ora.asm        ora.asm.type   ONLINE    ONLINE    rac01       
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac01       
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    rac01       
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac01       
ora.ons        ora.ons.type   ONLINE    ONLINE    rac01       
ora....SM1.asm application    ONLINE    ONLINE    rac01       
ora.rac01.gsd  application    OFFLINE   OFFLINE               
ora.rac01.ons  application    ONLINE    ONLINE    rac01       
ora.rac01.vip  ora....t1.type ONLINE    ONLINE    rac01       
ora....SM2.asm application    ONLINE    ONLINE    rac02       
ora.rac02.gsd  application    OFFLINE   OFFLINE               
ora.rac02.ons  application    ONLINE    ONLINE    rac02       
ora.rac02.vip  ora....t1.type ONLINE    ONLINE    rac02       
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac01