ORA-15186 ASMLIB ERROR FUNCTION = [ASM_OPEN], ERROR = [1], ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

Oracle Clusterware installed OK. ASM1 instance works OK while ASM2 would return ORA-15186/ORA-15063

SQL> startup pfile=/u01/app/oracle/product/10.2.0/asm/admin/+ASM/pfile/init.ora
ASM instance started
Total System Global Area 130023424 bytes
Fixed Size 2019032 bytes
Variable Size 102838568 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”

I changed permissions on RAC node 2 WYGORA02 as

[root@wygora02 ~]# chown oracle:dba /dev/mapper/mpath*
[root@wygora02 ~]# cd /dev/mapper/
[root@wygora02 mapper]# chmod 660 mpath*
[root@wygora02 mapper]# ls -l
total 0
crw——- 1 root root 10, 63 Feb 19 17:11 control
brw-rw—- 1 oracle dba 253, 7 Feb 19 17:11 mpath0
brw-rw—- 1 oracle dba 253, 8 Feb 19 17:11 mpath1
brw-rw—- 1 oracle dba 253, 9 Feb 19 17:11 mpath2
brw-rw—- 1 oracle dba 253, 10 Feb 19 17:11 mpath3
brw-rw—- 1 oracle dba 253, 11 Feb 19 17:11 mpath4
brw-rw—- 1 oracle dba 253, 12 Feb 19 17:11 mpath5
brw-rw—- 1 oracle dba 253, 13 Feb 19 17:11 mpath6
brw-rw—- 1 root disk 253, 0 Feb 19 17:11 VolGroup00-LogVol00
brw-rw—- 1 root disk 253, 6 Feb 19 17:11 VolGroup00-LogVol01
brw-rw—- 1 root disk 253, 1 Feb 19 17:11 VolGroup00-LogVol02
brw-rw—- 1 root disk 253, 3 Feb 19 17:11 VolGroup00-LogVol03
brw-rw—- 1 root disk 253, 2 Feb 19 17:11 VolGroup00-LogVol04
brw-rw—- 1 root disk 253, 5 Feb 19 17:11 VolGroup00-LogVol05
brw-rw—- 1 root disk 253, 4 Feb 19 17:11 VolGroup00-LogVol06

List ASM details at both servers

spool asm<#>.html
SET MARKUP HTML ON
set echo on
set pagesize 200
select * from v$asm_disk;
select * from v$asm_diskgroup;
select * from v$asm_operation;
select * from v$version;
show parameter asm
show parameter cluster
show parameter instance_type
show parameter instance_name
spool off
exit

2) Also, on each node please execute the next commands

$> /etc/init.d/oracleasm listdisks
$> ls -l /dev/oracleasm/disks/*
$> /etc/init.d/oracleasm querydisk <ASMLIB disk 1>
.
.
.
$> /etc/init.d/oracleasm querydisk <ASMLIB disk N>
# /usr/sbin/oracleasm-discover
# /usr/sbin/oracleasm-discover ‘ORCL:*’

3) Finally on each node we need to validate each ASMLIB disk has a valid ASM header thru kfed:

a) So first of all please compile kfed as follow (as Oracle user):
$> cd $ORACLE_HOME/rdbms/lib (ASM Oracle Home)
$> make -f ins_rdbms.mk ikfed

b) Verify kfed was copied to $ORACLE_HOME/bin (ASM Oracle Home)
$> ls -l $ORACLE_HOME/bin/kfed

c) Dump the first data block of each affected partition as follow:
$> $ORACLE_HOME/bin/kfed read <partition name> > /tmp/kfed<partition name>
Example:
$> $ORACLE_HOME/bin/kfed read /dev/oracleasm/disks/<ASMLIB disk 1> > /tmp/kfed_<ASMLIB disk 1>
.
$> $ORACLE_HOME/bin/kfed read /dev/oracleasm/disks/<ASMLIB disk N> > /tmp/kfed_<ASMLIB disk N>

Open kfed_log files and see the HEADER info if VALID

A healthy disk hearer will show “KFBTYP_DISKHEAD”
kfbh.endian:                          1 ; 0×000: 0×01
kfbh.hard:                          130 ; 0×001: 0×82
kfbh.type:                            1 ; 0×002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0×003: 0×01
kfbh.block.blk:                       0 ; 0×004: T=0 NUMB=0×0
kfbh.block.obj:              2147483648 ; 0×008: TYPE=0×8 NUMB=0×0
kfbh.check:                  1025391376 ; 0x00c: 0x3d1e3b10
kfbh.fcn.base:                        0 ; 0×010: 0×00000000

VOL1:
============================================================
kfbh.endian: 0 ; 0×000: 0×00
kfbh.hard: 0 ; 0×001: 0×00
kfbh.type: 0 ; 0×002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0×003: 0×00