11g Grid | Fixing Incident (BEA-101020 [HTTP]) detected in ..Middleware/gc_inst/user_projects

I have tonnes of these errors reported as critical at Grid Control. It is a bug, solved in 11.2 ( not released yet).

This error message is meaningless and can be safety ignored and there is one off patch to suppress this “error” Article ID 1139600.1

Log Details show:

<msg time=’2011-04-20T18:51:48.889+01:00′ org_id=’oracle’ comp_id=’ofm’
msg_id=’719226105′ type=’INCIDENT_ERROR’ level=’1′
host_id=’omsnode.oobm.travel.lcl’ host_addr=’10.241.156.201′ prob_key=’BEA-101020 [HTTP]‘
upstream_comp=” downstream_comp=” ecid=’0000Ixo2XQP2RPJ5Ing8yf1Dfim_00000M’
errid=’132′ detail_path=’/opt/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/incident/incdir_132′>
<txt>Errors in directory: /opt/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/incident/incdir_132  (incident=132):
null
</txt>

The incident doesn’t indicate cause of an error

Description
———–
Incident detected using watch rule “UncheckedException”:
Watch time:             Jan 20, 2011 6:51:48 PM BST
Watch ServerName:       EMGC_OMS1
Watch RuleType:         Log
Watch Rule:             (SEVERITY = ‘Error’) AND ((MSGID = ‘BEA-101020′) OR (MSGID = ‘BEA-101017′) OR (MSGID = ‘BEA-000802′))
Watch DomainName:       GCDomain
Watch Data:
DATE : Jan 20, 2011 6:51:48 PM BST
SERVER : EMGC_OMS1
MESSAGE : [ServletContext@2127000850[app:emgc module:/em path:/em spec-version:2.5]] Servlet failed with Exception
java.lang.IllegalStateException: Response already committed

To fix these issues follow this process : {From Metalink}

1.   Apply Patch 9882856 in the $AGENT_HOME monitoring the target for which the alert was raised.

2.    Create filter expression similar to the database 10g Alert Log Filter expression in Note 949858.1

.*BEA-(101020)\D.* – Any string
.*BEA-(101020)\D.* – Followed by the string â??BEA-â??
.*BEA-(101020)\D.* – Then 101020
.*BEA-(101020)\D.* – Followed by anything other than a digit
.*BEA-(101020)\D.* – Followed by any string

Which translates to the following errors:
BEA-101020
To add this filter expression, edit the AGENT_HOME/sysman/config/emd.properties file, add:
adrAlertLogAsErrorCodeExcludeRegex=.*BEA-(101020)\D.*
Note: There should be no spaces. This will make sure that the WLS incidents that match this regex are filtered.

3.  Restart agent

4. To manually clear the existing or old alerts about [..]EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml from Enterprise Manager Grid Control User Interface / webpage,
apply Patch 9914120

How to apply Patch 9882856?

[oracle@omsnode 9882856]$ opatch apply
Invoking OPatch 11.1.0.8.0
Oracle Interim Patch Installer version 11.1.0.8.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
Oracle Home       : /opt/app/oracle/product/11.2/db_1
Central Inventory : /opt/app/oracle/oraInventory
from           : /etc/oraInst.loc
OPatch version    : 11.1.0.8.0
OUI version       : 11.2.0.1.0
OUI location      : /opt/app/oracle/product/11.2/db_1/oui
Log file location : /opt/app/oracle/product/11.2/db_1/cfgtoollogs/opatch/opatch.log
Patch history file: /opt/app/oracle/product/11.2/db_1/cfgtoollogs/opatch/opatch_history.txt
OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogicServer”
ApplySession applying interim patch ’9882856′ to OH ‘/opt/app/oracle/product/11.2/db_1′
Running prerequisite checks…
Prerequisite check “CheckApplicable” failed.
The details are:
Patch 9882856: Required component(s) missing : [ oracle.sysman.top.agent, 11.1.0.1.0 ]
ApplySession failed during prerequisite checks: Prerequisite check “CheckApplicable” failed.
System intact, OPatch will not attempt to restore the system
OPatch failed with error code 74

[oracle@omsnode 9882856]$ which opatch
/opt/app/oracle/Middleware/WebLogicServer/agent11g/OPatch/opatch

Opatch failed as it is looking for ORACLE_HOME set to AGENT_HOME
[oracle@omsnode 9882856]$ export ORACLE_HOME=$AGENT_HOME

[oracle@omsnode 9882856]$ echo $ORACLE_HOME
/opt/app/oracle/Middleware/WebLogicServer/agent11g
[oracle@omsnode 9882856]$ opatch apply
Invoking OPatch 11.1.0.8.0
Oracle Interim Patch Installer version 11.1.0.8.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
Oracle Home       : /opt/app/oracle/Middleware/WebLogicServer/agent11g
Central Inventory : /opt/app/oracle/oraInventory
from           : /etc/oraInst.loc
OPatch version    : 11.1.0.8.0
OUI version       : 11.1.0.8.0
OUI location      : /opt/app/oracle/Middleware/WebLogicServer/agent11g/oui
Log file location : /opt/app/oracle/Middleware/WebLogicServer/agent11g/cfgtoollogs/opatch/opatch.log
Patch history file: /opt/app/oracle/Middleware/WebLogicServer/agent11g/cfgtoollogs/opatch/opatch_history.txt
OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogicServer”
ApplySession applying interim patch ’9882856′ to OH ‘/opt/app/oracle/Middleware/WebLogicServer/agent11g’
Running prerequisite checks…
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/opt/app/oracle/Middleware/WebLogicServer/agent11g’)
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch ’9882856′ for restore. This might take a while…
Backing up files affected by the patch ’9882856′ for rollback. This might take a while…
Patching component oracle.sysman.top.agent, 11.1.0.1.0…
Copying file to “/opt/app/oracle/Middleware/WebLogicServer/agent11g/sysman/admin/scripts/alertlogAdrAs.pl”
ApplySession adding interim patch ’9882856′ to inventory
Verifying the update…
Inventory check OK: Patch ID 9882856 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9882856 are present in Oracle Home.
The local system has been patched and can be restarted.
OPatch succeeded.

How to apply Patch 9914120?

[oracle@omsnode 9914120]$ opatch apply
Invoking OPatch 11.1.0.8.0
Oracle Interim Patch Installer version 11.1.0.8.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
Oracle Home       : /opt/app/oracle/Middleware/WebLogicServer/agent11g
Central Inventory : /opt/app/oracle/oraInventory
from           : /etc/oraInst.loc
OPatch version    : 11.1.0.8.0
OUI version       : 11.1.0.8.0
OUI location      : /opt/app/oracle/Middleware/WebLogicServer/agent11g/oui
Log file location : /opt/app/oracle/Middleware/WebLogicServer/agent11g/cfgtoollogs/opatch/opatch.log
Patch history file: /opt/app/oracle/Middleware/WebLogicServer/agent11g/cfgtoollogs/opatch/opatch_history.txt

OPatch detects the Middleware Home as “/opt/app/oracle/Middleware/WebLogicServer”
ApplySession applying interim patch ’9914120′ to OH ‘/opt/app/oracle/Middleware/WebLogicServer/agent11g’
Running prerequisite checks…
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/opt/app/oracle/Middleware/WebLogicServer/agent11g’)
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch ’9914120′ for restore. This might take a while…
Backing up files affected by the patch ’9914120′ for rollback. This might take a while…
Patching component oracle.sysman.top.agent, 11.1.0.1.0…
Copying file to “/opt/app/oracle/Middleware/WebLogicServer/agent11g/sysman/admin/metadata/weblogic_j2eeserver.xml”
ApplySession adding interim patch ’9914120′ to inventory
Verifying the update…
Inventory check OK: Patch ID 9914120 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9914120 are present in Oracle Home.
The local system has been patched and can be restarted.
OPatch succeeded.