Upload failed with ret=-10 / Agent is Running but Not Ready

Background : I have installed OMS/OMR and OMA successfully. In theory I should see OMA client successfully configured at OMS grid console but

$ ./emctl status agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.3.0
OMS Version       : 10.2.0.2.0
Protocol Version  : 10.2.0.2.0
Agent Home        : /opt/oracle/product/oracle10gAgent/agent10g
Agent binaries    : /opt/oracle/product/oracle10gAgent/agent10g
Agent Process ID  : 23314
Parent Process ID : 23295
—————————————————————
Agent is Running but Not Ready

To locate an exact error look at OMS files $OMS_HOME\sysman\log\emoms.log and $OMS_HOME\sysman\log\emoms.trc

My  $OMS_HOME\sysman\log\emoms.trc show error at  Trigger ‘SYSMAN.EM_VIOLATION_CHECKS’

2008-12-02 15:00:37,180 [SeverityLoad https://bmhp01:3872/emd/main/] ERROR XMLLoader.Handler processInsertException.462 – java.sql.SQLException: ORA-20612: Internal error processing severity. Closing availability record with negative duration for :  (target_guid = 1C2A2131D1A40C621929D7CE63032001) (collection_ts = 2008-12-02 14:47:07) (current_status = AGENT DOWN)
ORA-06512: at “SYSMAN.EM_SEVERITY”, line 2110
ORA-06512: at “SYSMAN.EM_SEVERITY”, line 3287
ORA-06512: at “SYSMAN.EM_VIOLATION_CHECKS”, line 791
ORA-04088: error during execution of trigger ‘SYSMAN.EM_VIOLATION_CHECKS’

Check at Google or metalink for further details on it.

- Sometimes you may have to delete records from OMR tables to fix this issue see Note:603011.1

- For ORA-04088: during execution of trigger SYSMAN.TARGETS_INSERT_TRIGGER , locate TARGET_GUID/AGENT_GUID at emoms.log  for example CE6B5C2E232766E01D3803EB03D70128 and run “delete from MGMT_TARGET_ASSOC_INSTANCE where target_guid = HEXTORAW(‘TARGET_GUID’);”

How to fix it? For clustered install (RAC/OS cluster) please see details here

$ ./emctl stop agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Stopping agent … stopped.

Delete all files in $AGENT_HOME/sysman/emd/upload and $AGENT_HOME/sysman/emd/state

$ ./emctl clearstate agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
EMD clearstate completed successfully

$ ./emctl unsecure agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Checking Agent for HTTP…   Done.
Agent is already stopped…   Done.
Unsecuring agent…   Started.
OMS Upload URL – http://ASST105157:4889/em/upload/ is locked or unavailable.

$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
EMD upload completed successfully

$ ./emctl status agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version     : 10.2.0.3.0
OMS Version       : 10.2.0.2.0
Protocol Version  : 10.2.0.2.0
Agent Home        : /opt/oracle/product/oracle10gAgent/agent10g
Agent binaries    : /opt/oracle/product/oracle10gAgent/agent10g
Agent Process ID  : 23314
Parent Process ID : 23295
Agent URL         : https://bmhp01:3872/emd/main/
Repository URL    : https://ASST105157:1159/em/upload
Started at        : 2008-12-02 14:52:07
Started by user   : oracle
Last Reload       : 2008-12-02 14:52:07
Last successful upload                       : 2008-12-02 15:05:03
Total Megabytes of XML files uploaded so far :     5.25
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    32.43%
Last successful heartbeat to OMS             : 2008-12-02 15:04:50
—————————————————————
Agent is Running and Ready

$ ./emctl secure agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Enter Agent Registration password :
Agent successfully stopped…   Done.
Securing agent…   Started.
Requesting an HTTPS Upload URL from the OMS…   Done.
Requesting an Oracle Wallet and Agent Key from the OMS…   Done.
Check if HTTPS Upload URL is accessible from the agent…   Done.
Configuring Agent for HTTPS in CENTRAL_AGENT mode…   Done.
EMD_URL set in /opt/oracle/product/oracle10gAgent/agent10g/sysman/config/emd.properties
Securing agent…   Successful.

I can see new new target at grid console