FAN (Fast Application Notification) & ONS (Oracle Notification Services)

Oracle Notification Services
ONS allows users to send SMS messages, e-mails, voice notifications, and fax messages in an easy-to-access manner. Oracle Clusterware uses ONS to send notifications about the state of the database instances to midtier applications that use this information for load-balancing and for fast failure detection. ONS is a daemon process that communicates with other ONS daemons
on other nodes which inform each other of the current state of the database components on the database server.

To add additional members or nodes that should receive notifications,  the hostname or IP address of the node should be added to the ons.config file.
The ONS configuration file is located in the $ORACLE_HOME/opmn/conf  directory and has the following format:
[oracle@oradb4 oracle]$ more $ORACLE_HOME/opmn/conf/ons.config
localport=6101
remoteport=6201
loglevel=3
useocr=on
nodes=oradb4.sumsky.net:6101,oradb2.sumsky.net:6201,
oradb1.sumsky.net:6201,oradb3.sumsky.net:6201,
onsclient1.sumsky.net:6200,onsclient2.sumsky.net:6200

The localport is the port that ONS binds to on the local host interface  to talk to local clients. The remoteport is the port that ONS binds to on all interfaces to talk to other ONS daemons. The loglevel indicates the  amount of logging that should be generated. Oracle supports logging levels from 1 through 9. ONS logs are generated in the $ORACLE_HOME/opmn/logs directory on the respective instances
The useocr parameter (valid values are on/off) indicates whether ONS should use the OCR to determine which instances and nodes are participating in the cluster. The nodes listed in the nodes line are all nodes in the network that will need to receive or send event notifications. This includes client machines where ONS is also running to receive FAN events for applications.

How do you monitor and track when these events are fired? Oracle has provided options where a script or utility or application (called server-side callouts), if placed in the $ORA_CRS_HOME/racg/usrco directory, will be executed automatically. For example, the following shell script, when placed in this directory will write out events generated by the Event Manager to the file defined by the symbol FAN_LOGFILE in the script:
[oracle@oradb4 oracle]$ more callout.sh
#! /bin/ksh
FAN_LOGFILE=/usr/app/oracle/product/10.2.0/crs/evm/admin/log/
oradb4_FANuptime.log
echo >> $FAN_LOGFILE &
Similarly, any scr