asm disk误设置pvid导致asm diskgroup无法mount恢复

数据库|mysql教程

asm,disk,设置,pvid,导致,diskgroup,

数据库-mysql教程

thinkphp sql 源码,vscode 状态栏颜色,bios引导ubuntu,tomcat的具体安装,爬虫工具web,linux php 缓存,海南策划型seo提高口碑,多套网站程序会员系统打通,帝国免费商城模板下载地址lzw

有朋友找到我说他们把以前存储到AIX直连的存储切换为含光纤交换机的存储网络后,RAC无法启动,让我给予支持.通过分析是由于换链路之后开始磁盘顺序不对,维护人员对其asm disk 设置了pvid,导致asm 磁盘组无法正常mount,从而使得含votedisk的dg的asm disk无法正

易语言快递打印源码,vscode设置背景色,ubuntu打开蓝牙,tomcat运行权限不够,魚佬爬虫,php 输出tab,seo点击工具是真的吗,网站模板超市源码,小俊模板打包lzw

电台 网站程序源码,vscode多列粘贴,ubuntu占用硬盘,tomcat网页重定向,sqlite3怎么查看表头,微信小程序网课能爬虫吗,php 运行exe,普陀seo优化哪家好,偷网站css,个人博客模板htmllzw

有朋友找到我说他们把以前存储到AIX直连的存储切换为含光纤交换机的存储网络后,RAC无法启动,让我给予支持.通过分析是由于换链路之后开始磁盘顺序不对,维护人员对其asm disk 设置了pvid,导致asm 磁盘组无法正常mount,从而使得含votedisk的dg的asm disk无法正常访问,从而RAC的cssd进程无法启动,同样数据文件的磁盘组也无法mount,通过kfed修复成功,实现数据0丢失.

平台版本信息(2节点RAC)

$ sqlplus -vSQL*Plus: Release 11.2.0.4.0 Production$ uname -aAIX db2 1 7 00F9733E4C00

GI日志报错信息

-12-20 16:44:08.769: [ohasd(6946818)]CRS-2769:Unable to failover resource ora.diskmon.-12-20 16:44:11.775: [cssd(9502756)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/db1/cssd/ocssd.log-12-20 16:44:26.791: [cssd(9502756)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; 、Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/db1/cssd/ocssd.log-12-20 16:44:41.812: [cssd(9502756)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/db1/cssd/ocssd.log

从这里可以看出来是由于RAC启动过程中无法获得votedisk使得其无法正常启动,通过分析日志找出来votedisk相关磁盘

-12-15 17:36:15.424: [cssd(10027070)]CRS-1605:CSSD voting file is online: /dev/rhdisk4; details in /u01/app/11.2.0/grid/log/db1/cssd/ocssd.log-12-15 17:36:15.433: [cssd(10027070)]CRS-1605:CSSD voting file is online: /dev/rhdisk5; details in /u01/app/11.2.0/grid/log/db1/cssd/ocssd.log-12-15 17:36:15.445: [cssd(10027070)]CRS-1605:CSSD voting file is online: /dev/rhdisk6; details in /u01/app/11.2.0/grid/log/db1/cssd/ocssd.log

从这里可以知道rhdisk4,5,6为votedisk对应磁盘,使用kfed查看磁盘头信息

$kfed read /dev/rhdisk4kfbh.endian: 201 ; 0x000: 0xc9kfbh.hard:194 ; 0x001: 0xc2kfbh.type:212 ; 0x002: *** Unknown Enum ***kfbh.datfmt: 193 ; 0x003: 0xc1kfbh.block.blk: 0 ; 0x004: blk=0kfbh.block.obj: 0 ; 0x008: file=0kfbh.check: 0 ; 0x00c: 0x00000000kfbh.fcn.base: 0 ; 0x010: 0x00000000kfbh.fcn.wrap: 0 ; 0x014: 0x00000000kfbh.spare1:0 ; 0x018: 0x00000000kfbh.spare2:0 ; 0x01c: 0x000000001102BEE00 C9C2D4C1 00000000 00000000 00000000 [................]1102BEE10 00000000 00000000 00000000 00000000 [................] Repeat 6 times1102BEE80 00F9733D 67553E0A 00000000 00000000 [..s=gU>.........]1102BEE90 00000000 00000000 00000000 00000000 [................] Repeat 246 timesKFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][212]$kfed read /dev/rhdisk4 blkn=1kfbh.endian:0 ; 0x000: 0x00kfbh.hard:130 ; 0x001: 0x82kfbh.type: 2 ; 0x002: KFBTYP_FREESPCkfbh.datfmt:2 ; 0x003: 0x02kfbh.block.blk: 1 ; 0x004: blk=1kfbh.block.obj: 2147483648 ; 0x008: disk=0kfbh.check: 3883664132 ; 0x00c: 0xe77c0304kfbh.fcn.base: 0 ; 0x010: 0x00000000kfbh.fcn.wrap: 0 ; 0x014: 0x00000000kfbh.spare1:0 ; 0x018: 0x00000000kfbh.spare2:0 ; 0x01c: 0x00000000kfdfsb.aunum: 0 ; 0x000: 0x00000000kfdfsb.max: 254 ; 0x004: t:23 ; 0x006: 0x0017kfdfsb.bound: 0 ; 0x008: 0x0000kfdfsb.flag:1 ; 0x00a: B=1kfdfsb.ub1spare:0 ; 0x00b: 0x00kfdfsb.spare[0]:0 ; 0x00c: 0x00000000kfdfsb.spare[1]:0 ; 0x010: 0x00000000kfdfsb.spare[2]:0 ; 0x014: 0x00000000kfdfse[0].fse:119 ; 0x018: FREE=0x7 FRAG=0x7kfdfse[1].fse: 16 ; 0x019: FREE=0x0 FRAG=0x1…………$kfed read /dev/rhdisk4 blkn=510kfbh.endian:0 ; 0x000: 0x00kfbh.hard:130 ; 0x001: 0x82kfbh.type: 1 ; 0x002: KFBTYP_DISKHEADkfbh.datfmt:1 ; 0x003: 0x01kfbh.block.blk:254 ; 0x004: blk=254kfbh.block.obj: 2147483648 ; 0x008: disk=0kfbh.check: 3460116983 ; 0x00c: 0xce3d31f7kfbh.fcn.base: 0 ; 0x010: 0x00000000kfbh.fcn.wrap: 0 ; 0x014: 0x00000000kfbh.spare1:0 ; 0x018: 0x00000000kfbh.spare2:0 ; 0x01c: 0x00000000kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000kfdhdb.driver.reserved[5]: 0 ; 0x01c: pat:186646528 ; 0x020: 0x0b200000kfdhdb.dsknum: 0 ; 0x024: 0x0000kfdhdb.grptyp: 2 ; 0x026: KFDGTP_NORMALkfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBERkfdhdb.dskname:CRS_0000 ; 0x028: length=8kfdhdb.grpname:CRS ; 0x048: length=3kfdhdb.fgname: CRS_0000 ; 0x068: length=8…………

由上述分析可以基本上确定是asm disk header 被破坏,进一步分析破坏原因

[db2/dev#]lspvhdisk000f9733ef7cf27e9 rootvgactivehdisk100f9733e21b953e6 rootvgactivehdisk200f9733e21b97a83 appvg activehdisk300f9733e21b98434 appvg activehdisk400f9733d67553e0a None hdisk500f9733d67553f31 None hdisk600f9733d67554011 None hdisk700f9733d67554165 None hdisk800f9733d675541e5 None hdisk900f9733d675542e4 None hdisk10 noneNone [db2/dev#]ls -l rhdisk*crw------- 2 rootsystem 24, 1 Oct 18 11:45 rhdisk0crw------- 1 rootsystem 24, 3 Oct 18 13:27 rhdisk1crw------- 1 rootsystem 24, 5 Dec 20 20:02 rhdisk10crw------- 1 rootsystem 24, 2 Oct 18 13:32 rhdisk2crw------- 1 rootsystem 24, 0 Oct 18 13:32 rhdisk3crw-rw---- 1 gridasmadmin24, 8 Dec 20 20:02 rhdisk4crw-rw---- 1 gridasmadmin24, 9 Dec 20 20:02 rhdisk5crw-rw---- 1 gridasmadmin24, 10 Dec 20 20:02 rhdisk6crw-rw---- 1 gridasmadmin24, 4 Dec 20 20:02 rhdisk7crw-rw---- 1 gridasmadmin24, 6 Dec 20 20:02 rhdisk8crw-rw---- 1 gridasmadmin24, 7 Dec 20 20:02 rhdisk9

从这里基本上可以看出来,是由于磁盘头被重写了pvid,导致asm disk header 被破坏.进一步分析asm log,确定哪些磁盘被用作asm disk

SQL> CREATE DISKGROUP CRS NORMAL REDUNDANCY DISK /dev/rhdisk4,/dev/rhdisk5,/dev/rhdisk6 ATTRIBUTE compatible.asm=11.2.0.0.0,au_size=1M /* ASMCA */ NOTE: Assigning number (1,0) to disk (/dev/rhdisk4)NOTE: Assigning number (1,1) to disk (/dev/rhdisk5)NOTE: Assigning number (1,2) to disk (/dev/rhdisk6)NOTE: initializing header on grp 1 disk CRS_0000NOTE: initializing header on grp 1 disk CRS_0001NOTE: initializing header on grp 1 disk CRS_0002SQL> CREATE DISKGROUP DATA EXTERNAL REDUNDANCY DISK /dev/rhdisk9 SIZE 614400M ATTRIBUTE compatible.asm=11.2.0.0.0,au_size=1M /* ASMCA */ NOTE: Assigning number (2,0) to disk (/dev/rhdisk9)NOTE: initializing header on grp 2 disk DATA_0000SQL> CREATE DISKGROUP FBA EXTERNAL REDUNDANCY DISK /dev/rhdisk8 SIZE 204800M ATTRIBUTE compatible.asm=11.2.0.0.0,au_size=1M /* ASMCA */ NOTE: Assigning number (3,0) to disk (/dev/rhdisk8)NOTE: initializing header on grp 3 disk FBA_0000SQL> CREATE DISKGROUP ARCH EXTERNAL REDUNDANCY DISK /dev/rhdisk7 SIZE 102400M ATTRIBUTE compatible.asm=11.2.0.0.0,au_size=1M /* ASMCA */ NOTE: Assigning number (4,0) to disk (/dev/rhdisk7)NOTE: initializing header on grp 4 disk ARCH_0000

这里可以确定asm disk为rhdisk[4-9],通过kfed分析全部和rhdisk4一样的问题,也符合lspv查询出来的结果,使用kfed repair修复asm disk header后

SQL> alter diskgroup data mount;Diskgroup altered.SQL> alter diskgroup fba mount;Diskgroup altered.SQL> alter diskgroup arch mount;Diskgroup altered.SQL> alter diskgroup crs mount;Diskgroup altered.SQL> select group_number,disk_number,path from v$asm_disk;GROUP_NUMBER DISK_NUMBER PATH------------ ----------- -------------------------------------------------- 2 0 /dev/rhdisk4 2 1 /dev/rhdisk5 2 2 /dev/rhdisk6 1 0 /dev/rhdisk7 4 0 /dev/rhdisk8 3 0 /dev/rhdisk96 rows selected.SQL> select group_number,name from v$asm_diskgroup;GROUP_NUMBER NAME------------ ------------------------------------------------------------ 1 ARCH 2 CRS 3 DATA 4 FBA

这里证明通过kfed对磁盘头的修复,asm磁盘组已经全部mount成功,GI状态也恢复正常

[db2/#]crsctl status res -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.ARCH.dgONLINE ONLINE db1ONLINE ONLINE db2ora.CRS.dgONLINE ONLINE db1ONLINE ONLINE db2ora.DATA.dgONLINE ONLINE db1ONLINE ONLINE db2ora.FBA.dgONLINE ONLINE db1ONLINE ONLINE db2ora.LISTENER.lsnrONLINE ONLINE db1ONLINE ONLINE db2ora.asmONLINE ONLINE db1Started ONLINE ONLINE db2Started ora.gsdOFFLINE OFFLINEdb1OFFLINE OFFLINEdb2workONLINE ONLINE db1ONLINE ONLINE db2ora.onsONLINE ONLINE db1ONLINE ONLINE db2ora.registry.acfsONLINE ONLINE db1ONLINE ONLINE db2--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr1 ONLINE ONLINE db1ora.cvu1 ONLINE ONLINE db1ora.db1.vip1 ONLINE ONLINE db1ora.db2.vip1 ONLINE ONLINE db2ora.nkora.db1 ONLINE ONLINE db1Open2 ONLINE ONLINE db2Openora.oc4j1 ONLINE ONLINE db1ora.scan1.vip1 ONLINE ONLINE db1

这里忽略了一个问题,在修复磁盘头之前没有清除pvid,导致磁盘头修复后,pvid依然存储在odm中

[db2/dev#]lspvhdisk000f9733ef7cf27e9 rootvgactivehdisk100f9733e21b953e6 rootvgactivehdisk200f9733e21b97a83 appvg activehdisk300f9733e21b98434 appvg activehdisk400f9733d67553e0a None hdisk500f9733d67553f31 None hdisk600f9733d67554011 None hdisk700f9733d67554165 None hdisk800f9733d675541e5 None hdisk900f9733d675542e4 None hdisk10 noneNone

通过分析发现fba磁盘组中无任何记录,使用该磁盘组进行直接清除pvid测试

$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Sun Dec 21 03:13:31 Copyright (c) 1982, , Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsSQL> alter diskgroup fba dismount;Diskgroup altered.SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management options$ exitYou have mail in /usr/spool/mail/root[db2/#]chdev -l hdisk8 -a pv=clearhdisk8 changed[db2/#]lspvhdisk000f9733ef7cf27e9 rootvgactivehdisk100f9733e21b953e6 rootvgactivehdisk200f9733e21b97a83 appvg activehdisk300f9733e21b98434 appvg activehdisk400f9733d67553e0a None hdisk500f9733d67553f31 None hdisk600f9733d67554011 None hdisk700f9733d67554165 None hdisk8noneNone hdisk900f9733d675542e4 None hdisk10 noneNone [db2/#]su - grid$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Sun Dec 21 03:15:19 Copyright (c) 1982, , Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsSQL> alter diskgroup fba mount;Diskgroup altered.SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management options

通过测试直接清除pvid asm 磁盘头依然工作正常,关闭GI,使用chdev清除hdisk[4-9]所有pvid,启动GI一切正常

[db1/#]crsctl status res -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.ARCH.dgONLINE ONLINE db1ONLINE ONLINE db2ora.CRS.dgONLINE ONLINE db1ONLINE ONLINE db2ora.DATA.dgONLINE ONLINE db1ONLINE ONLINE db2ora.FBA.dgONLINE ONLINE db1ONLINE ONLINE db2ora.LISTENER.lsnrONLINE ONLINE db1ONLINE ONLINE db2ora.asmONLINE ONLINE db1Started ONLINE ONLINE db2Started ora.gsdOFFLINE OFFLINEdb1OFFLINE OFFLINEdb2workONLINE ONLINE db1ONLINE ONLINE db2ora.onsONLINE ONLINE db1ONLINE ONLINE db2ora.registry.acfsONLINE ONLINE db1ONLINE ONLINE db2--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr1 ONLINE ONLINE db1ora.cvu1 ONLINE ONLINE db1ora.db1.vip1 ONLINE ONLINE db1ora.db2.vip1 ONLINE ONLINE db2ora.nkora.db1 ONLINE ONLINE db1Open2 ONLINE ONLINE db2Openora.oc4j1 ONLINE ONLINE db1ora.scan1.vip1 ONLINE ONLINE db1[db1/#]lspvhdisk000f9733df7c7a9db rootvgactivehdisk100f9733d21dad8fe rootvgactivehdisk200f9733d21dbd08b appvg activehdisk300f9733d21dbd2ab appvg activehdisk4noneNone hdisk5noneNone hdisk6noneNone hdisk7noneNone hdisk8noneNone hdisk9noneNone hdisk10 noneNone

至此设置pvid导致asm disk header损坏的asm 恢复正常,实现数据0丢失。

温馨提示:aix asm disk磁盘中不能设置pvid,否则将会导致asm disk header 损坏,无法正常mount

使用asm disk header 自动备份信息恢复异常asm disk headeroracle 12.1 RAC的ocr磁盘组异常恢复ASM DISK HEADER 备份与恢复手工修复ASM DISK HEADER 异常ORACLE 12C RAC修改ocr/votedisk/asm spfile所在磁盘组名称因asm sga_target设置不当导致11gr2 rac无法正常启动ORACLE 12C RAC hub AND leaf 相互转换网关不通致使vip/lsnr资源异常

原文地址：asm disk误设置pvid导致asm diskgroup无法mount恢复, 感谢原作者分享。