今天准备用自己的rac虚拟机做点实验的，可是虚拟机拉起来时，频频报错，估计好久不用了（前一阵子有厂商来推销虚拟化全在那上做测试了），挨个检查发现是OCR出故障了，没办法重建下吧，顺便记录下重建的过程
专业领域包括网站设计制作、成都网站建设、商城网站定制开发、微信营销、系统平台开发， 与其他网站设计及系统开发公司不同，成都创新互联公司的整合解决方案结合了帮做网络品牌建设经验和互联网整合营销的理念，并将策略和执行紧密结合，为客户提供全网互联网整合方案。
1.使用orccheck命令报错
[root@rac1 ~]#ocrcheck
PROT-602: Failedto retrieve datafrom the cluster registry
PROC-26: Errorwhile accessing the
physical storage
2.使用ocrconfig-restore恢复仍然报错
[root@rac1 rac-cluster]#ocrconfig-restore/u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
PROT-35: The configured OCR locations arenot accessible.
3.强制停掉两个节点的crs（两个节点都要执行）
[root@rac1 rac-cluster]#crsctl stop crs-f
CRS-2791:Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673:Attempting to stop 'ora.mDNSd' on 'rac1'
CRS-2673:Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673:Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673:Attempting to stop 'ora.asm' on 'rac1'
CRS-2677:Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677:Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677:Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677:Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677:Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677:Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.crf' on 'rac1'
CRS-2677:Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677:Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677:Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793:Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133:Oracle High Availability
Services has been stopped.
[root@rac2 ~]#/u01/app/11.2.0/grid/bin/crsctl stop crs-f
CRS-2791:Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673:Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2673:Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2673:Attempting to stop 'ora.evmd' on 'rac2'
CRS-2673:Attempting to stop 'ora.asm' on 'rac2'
CRS-2677:Stop of 'ora.evmd' on 'rac2' succeeded
CRS-2677:Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677:Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2677:Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673:Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2677:Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2673:Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677:Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673:Attempting to stop 'ora.crf' on 'rac2'
CRS-2677:Stop of 'ora.crf' on 'rac2' succeeded
CRS-2673:Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677:Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673:Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677:Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793:Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133:Oracle High Availability
Services has been stopped.
4.在节点一上已独占方式启动(只启动ASM实例但不启动CRS)
[root@rac1 ~]#crsctl start crs-excl-nocrs
CRS-4123:Oracle High Availability Services has been started.
CRS-2672:Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676:Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672:Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676:Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672:Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672:Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676:Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676:Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672:Attempting to start 'ora.cssd' on 'rac1'
CRS-2672:Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676:Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676:Start of 'ora.cssd' on 'rac1' succeeded
CRS-2679:Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672:Attempting to start 'ora.ctssd' on 'rac1'
CRS-2681:Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672:Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676:Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676:Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672:Attempting to start 'ora.asm' on 'rac1'
CRS-2676:Start of 'ora.asm' on 'rac1'
succeeded
5.重建原votedisk所在的磁盘组
[root@rac1 ~]# su- grid
[grid@rac1 ~]$ sqlplus/as sysasm
SQL*Plus: Release11.2.0.4.0 Productionon Sun Feb2800:09:322016
Copyright (c)1982,2013, Oracle. All rights reserved.
Connectedto:
OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production
With theReal Application Clustersand Automatic Storage Management options
SQL>create diskgroup systemdg normal redundancydisk'/dev/asm-diskb','/dev/asm-diskc','/dev/asm-diskd' ATTRIBUTE'compatible.rdbms'='11.2','compatible.asm'='11.2';
Diskgroup created.
6.使用最新的OCR备份进行恢复
[root@rac1 ~]#ocrconfig-restore/u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
[root@rac1 ~]#ocrcheck
Statusof Oracle Cluster Registryisas follows :
Version : 3
Totalspace (kbytes) : 262120
Usedspace (kbytes) : 2900
Availablespace (kbytes) : 259220
ID :1207909490
Device/File Name : +SYSTEMDG
Device/File integritycheck succeeded
Device/Filenot configured
Device/Filenot configured
Device/Filenot configured
Device/Filenot configured
Cluster registry integritycheck succeeded
Logical corruptioncheck succeeded
7.恢复votedisk
[root@rac1 ~]#crsctl query css votedisk
Located0 votingdisk(s).
[grid@rac1 ~]$ crsctl replace votedisk +SYSTEMDG
Successful additionof votingdisk a0e3b9c0c5934f79bf67aa783618b59d.
Successful additionof votingdisk 422a673bdbc94f97bff51f66f84a105e.
Successful additionof votingdisk a1b110709eb84f39bf351f1b01aa0c7d.
Successfully replaced votingdiskgroupwith+SYSTEMDG.
CRS-4266: Votingfile(s) successfully replaced
[root@rac1 rac-cluster]#crsctl query css votedisk
## STATE File Universal Id File NameDiskgroup
-- ----- ----------------- --------- ---------
1. ONLINE 03bd9e6851cf4fb4bf4c92e24a3d71cc (/dev/asm-diskb)[SYSTEMDG]
2. ONLINE 362f2cfb5b3d4f29bf319c668e0efbe4 (/dev/asm-diskc)[SYSTEMDG]
3. ONLINE
57659c53f4284fbdbfeabafb20c3fbdd (/dev/asm-diskd)[SYSTEMDG]
Notes：如果在恢复时遇到如下错误：
[grid@rac1 ~]$crsctlreplace votedisk +SYSTEMDG
CRS-4602: Failed27toadd votingfile a0e3b9c0c5934f79bf67aa783618b59d.
CRS-4602: Failed27toadd votingfile 422a673bdbc94f97bff51f66f84a105e.
CRS-4602: Failed27toadd votingfile a1b110709eb84f39bf351f1b01aa0c7d.
Failedtoreplace votingdiskgroupwith+SYSTEMDG.
CRS-4000: CommandReplace failed,or completedwith errors.
--登陆grid用户，检查并修改asm_diskstring参数
SQL>alter systemset asm_diskstring='/dev/asm*';
System altered.
--检查并修改spfile参数，并重启ASM
SQL>create spfilefrom memory;
File created.
8.重启crs
[root@rac1 ~]#crsctl stop crs
CRS-2791:Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673:Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673:Attempting to stop 'ora.asm' on 'rac1'
CRS-2673:Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677:Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677:Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677:Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2677:Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677:Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677:Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673:Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677:Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793:Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133:Oracle High Availability Services has been stopped.
[root@rac1 ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
9.由于启动较慢，过几分钟查看状态
[root@rac1 ~]# crs_stat-t
Name Type Target State Host
------------------------------------------------------------
ora.DATADG.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac1
ora....EMDG.dg ora....up.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.orcl.db ora....se.type ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora.rac2.vip ora....t1.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type ONLINE ONLINE rac1
[root@rac1 ~]#crsctlcheck crs
CRS-4638: Oracle High Availability Servicesis online
CRS-4537: Cluster Ready Servicesis online
CRS-4529: Cluster Synchronization Servicesis online
CRS-4533: Event Manageris online
10.但是在启动db的时候报错了。。。
PRCR-1079 : Failedto start resource ora.orcl.db
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-00205: errorin identifying controlfile,check alertlogfor more info
.For details referto "(:CLSN00107:)"in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-00205: errorin identifying controlfile,check alertlogfor more info
.For details referto "(:CLSN00107:)"in "/u01/app/11.2.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674:Start of 'ora.orcl.db' on 'rac1' failed
CRS-2674:Start of 'ora.orcl.db' on 'rac2' failed
CRS-2632:There are no more
servers to try to place resource 'ora.orcl.db' on that would satisfy its
placement policy
--查看了下trace发现SYSTEMDG中的控制文件没了，装RAC的时候就没规划好，导致控制文件放到了votedisk里，但是SYSTEMDG刚刚被重建了，所以丢了
ALTER SYSTEMSET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.8.222)(PORT=1521))' SCOPE=MEMORY SID='orcl1';
ALTERDATABASE MOUNT/* db agent *//* {1:40511:558} */
This instance was firstto mount
NOTE: Loaded library: System
SUCCESS: diskgroup DATADG was mounted
SUCCESS: diskgroup SYSTEMDG was mounted
NOTE: dependencybetweendatabase orcland diskgroup resource ora.DATADG.dgis established
ORA-00210: cannotopen the specified controlfile
ORA-00202: controlfile:'+SYSTEMDG/orcl/controlfile/current.261.904952269'
ORA-17503: ksfdopn:2 Failedtoopenfile+SYSTEMDG/orcl/controlfile/current.261.904952269
ORA-15012: ASMfile'+SYSTEMDG/orcl/controlfile/current.261.904952269' doesnot exist
ORA-205 signalled during:ALTERDATABASE MOUNT/* db agent *//* {1:40511:558} */...
Sat Feb2723:48:142016
Shutting down instance (abort)
License high water mark=1
USER (ospid:21150): terminating the instance
Instance terminatedbyUSER, pid=21150
Sat Feb2723:48:142016
Instanceshutdown complete
11.将数据库启动到nomount阶段
[grid@rac1 trace]$srvctl startdatabase-d orcl-o nomount
12.在节点一上使用rman恢复一份控制文件到SYSTEMDG上
[oracle@rac1 ~]$ rman target/
Recovery Manager: Release11.2.0.4.0- Productionon Sat Feb2723:53:352016
Copyright (c)1982,2011, Oracleand/or its affiliates. All rights reserved.
connectedto targetdatabase: ORCL (not mounted)
RMAN>restore controlfileto'+SYSTEMDG'from'+DATADG/orcl/controlfile/current.260.901699963';
Startingrestore at2016/02/2723:53:43
using targetdatabase controlfile insteadof recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=34 instance=orcl1 device type=DISK
channel ORA_DISK_1: copied controlfile copy
Finishedrestore at2016/02/2723:53:52
13.已恢复好的控制文件路径和名称
ASMCMD> pwd
+systemdg/orcl/CONTROLFILE
ASMCMD> ls
current.261.904953225
14.节点一上登陆db，修改control_files参数
[oracle@rac1 ~]$ sqlplus/as sysdba
SQL*Plus: Release11.2.0.4.0 Productionon Sat Feb2723:55:092016
Copyright (c)1982,2013, Oracle. All rights reserved.
Connectedto:
OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production
With the Partitioning,Real Application Clusters, Automatic Storage Management, OLAP,
Data MiningandReal Application Testing options
SQL>alter systemset control_files='+DATADG/orcl/controlfile/current.260.901699963','+SYSTEMDG/orcl/CONTROLFILE/current.261.904953225' scope=spfile sid='*';
System altered.
15.重启数据库，并检查数据库状态
[grid@rac1 trace]$srvctl stopdatabase-d orcl
[grid@rac1 trace]$srvctl startdatabase-d orcl
[grid@rac1 trace]$srvctl statusdatabase-d orcl
Instance orcl1is runningon node rac1
Instance orcl2is runningon node rac2
16.保险起见，查看两个节点上的控制文件参数是否正确
[oracle@rac1 ~]$ sqlplus/as sysdba
SQL*Plus: Release11.2.0.4.0 Productionon Sat Feb2723:58:472016
Copyright (c)1982,2013, Oracle. All rights reserved.
Connectedto:
OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production
With the Partitioning,Real Application Clusters, Automatic Storage Management, OLAP,
Data MiningandReal Application Testing options
SQL> show parameter control_file
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------------------------------------------------------------------------
control_file_record_keep_time integer 7
control_files string +DATADG/orcl/controlfile/current.260.901699963,+SYSTEMDG/orcl/controlfile/current.261.904953225
[oracle@rac2 ~]$ sqlplus/as sysdba
SQL*Plus: Release11.2.0.4.0 Productionon Sat Feb2723:59:292016
Copyright (c)1982,2013, Oracle. All rights reserved.
Connectedto:
OracleDatabase 11g Enterprise Edition Release11.2.0.4.0- 64bit Production
With the Partitioning,Real Application Clusters, Automatic Storage Management, OLAP,
Data MiningandReal Application Testing options
SQL> show parameter control_file
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------------------------------------------------------------------------
control_file_record_keep_time integer 7
control_files string +DATADG/orcl/controlfile/current.260.901699963,+SYSTEMDG/orcl/controlfile/current.261.904953225
OK，收工~~~~
重庆分公司
重庆分公司
