Upload failed with ret=-10 / Agent is Running but Not Ready : Clustered node install

Problem Statement I am configuring Oracle Agent for a grid console. My system is HP TRUE64 and /opt/oracle/Agent10g is a clustered file system like QFS/GFS/OCFS. A clustered file system is shared between 2 nodes, in my case $ORACLE_HOME is shared between cluster1 and cluster2.

I could get number of standalone agents working except this one because it is on a clustered file system.

Agents are on a clustered file system so thinking each one of them need separate config files , I made 2 separate copies of most configuration files
- emd.properties
- /opt/oracle/product/agent10g/sysman/config/b64LocalCertificate.txt
-/opt/oracle/product/agent10g/sysman/config/b64InternetCertificate.txt
Also set a different directory on each server for
- emdWalletDest=/opt/oracle/product/agent10g/sysman/config/server
I could connect to Agent URL : http://Agent:3872/emd/main/ & Repository URL : http://oms:4889/em/upload/ successfully.

Result : When I start Agent on one machine , the agent running on other machine shuts down. I almost decided to reinstall individual agent but then came across oracle documentation  “Reconfiguring a Standalone Agent to an Oracle RAC Agent”
Reconfiguration of a standalone agent occurs when you want to configure this agent (with standalone configurations) as a Oracle RAC agent.
To reconfigure a standalone agent as a Oracle RAC agent, you must execute the agentca script with the following options:
<Agent_Home>/bin/agentca -f -c “node1,node2….” [-t -i oraInstloc -n CLUSTER_NAME ]

cluster1 (unknown)> ./agentca -f -c “cluster1 ,cluster2″

Stopping the agent using /opt/oracle/product/agent10g/bin/emctl  stop agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
Stopping agent … stopped.
Running agentca using /opt/oracle/product/agent10g/oui/bin/runConfig.sh ORACLE_HOME=/opt/oracle/product/agent10g ACTION=Configure MODE=Perform RESPONSE_FILE=/opt/oracle/product/agent10g/response_file RERUN=TRUE INV_PTR_LOC=/var/opt/oracle/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform – mode is starting for action: Configure
You can see the log file: /opt/oracle/product/agent10g/cfgtoollogs/oui/configActions2008-11-27_04-54-03-PM.log
ERROR: Unable to convert from “UTF-8″ to “ISO8859-1″ for NLS!
ÿÿÿÿÿÿÿÿ
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /opt/oracle/oraInventory
‘UpdateNodeList’ was successful.
Starting the agent using /opt/oracle/product/agent10g/bin/emctl  start agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
Starting agent …………………………… started but not ready.

After creating clustered agent,  Cluster1 agent started OK. I could locate cluster1 under Targets on my grid control. But there is still an issue with cluster2 node

clsueter2(unknown)> ./emctl start agent
EM Configuration issue. /opt/oracle/product/agent10g/cluster2/sysman/config/emd.properties not found.

A closer look revealed that after converting agent into clustered config, default directories picked up for configuration are  different
For node 1 – $AGENT_HOME/cluster1/sysman/config/ & for node 2 $AGENT_HOME/cluster2/sysman/config/  changed from $AGENT_HOME/sysman/config/

I thought this could be resolved by 2 methods

• Copying the directory $AGENT_HOME/cluster1 as $AGENT_HOME/cluster2 , Edit “emd.properties“ and set EMD_URL as per machines

OK, I copied directories and started agent at cluster1, went OK. I then started agent at cluster2, went well as well but moments later Agent at Cluster 1 stopped. It revealed that they were sharing same port 3872 and only 1 agent could be active on that port. I then changed ports at emd.properties but still agents won’t start

cluster1/cluster2 > ./emctl start agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
Starting agent …………………………… failed.
Consult the log files in: /opt/oracle/product/agent10g/cluster1/sysman/log

I finally decided to : Run “agentca” at cluster2 node as well and oracle will create directory and files for node 2 at $AGENT_HOME/cluster2/sysman/config/

cluster1(unknown)> rm –Rf $AGENT_HOME/cluster2

Cluster2 (unknown)> ./agentca -f -c “cluster1, cluster2″

Cluster2 (unknown)> ./emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.2.0
OMS Version       : 10.2.0.4.0
Protocol Version  : 10.2.0.2.0
—————————————————————
Agent is Running but Not Ready

Cluster2 (unknown)> ./emctl upload agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
—————————————————————
upload failed with ret=-10

Run following steps to push agent in “Agent is Running and Ready” status

$ ./emctl stop agent

Delete all files at $AGENT_HOME/sysman/emd/upload and $AGENT_HOME/sysman/emd/state
export AGENT_HOME=/opt/oracle/product/agent10g/
echo $AGENT_HOME
rm –Rf $AGENT_HOME/sysman/emd/upload/*.*
rm –Rf $AGENT_HOME/sysman/emd/state/*.*

Since this is clustered install , directories will be different for each node
rm –Rf /opt/oracle/product/agent10g/<cluster 1>/sysman/emd/upload/*.*
rm –Rf /opt/oracle/product/agent10g/<cluster 1>/sysman/emd/state/*.*
rm –Rf /opt/oracle/product/agent10g/<cluster 2>/sysman/emd/upload/*.*
rm –Rf /opt/oracle/product/agent10g/<cluster 2>/sysman/emd/state/*.*

$ ./emctl clearstate agent
$ ./emctl unsecure agent
$ ./emctl start agent
$ ./emctl upload agent
$ ./emctl secure agent

cluster2(unknown)> ./emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.2.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
—————————————————————
Last Reload       : 2008-12-04 16:44:26
Last successful upload                       : 2008-12-04 16:46:11
Total Megabytes of XML files uploaded so far :     0.58
Number of XML files pending upload           :        2
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    19.55%
Last successful heartbeat to OMS             : 2008-12-04 16:45:33
—————————————————————
Agent is Running and Ready

Both systems are communicating with grid now