Error while starting CommonDomain_webtier in fusion applications

This error may happen if there are some residual processes running on accidental failure of the services.

Symptoms

[oracle@fusionapp bin]$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
================================================================================
opmn id=fusionapp.chainsys.com:12043
Response: 0 of 1 processes started.

ias-instance id=CommonDomain_webtier
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
  ohs1/OHS/OHS/

Error
--> Process (index=1,uid=568205325,pid=10291)
  failed to start a managed process after the maximum retry limit
  Log:
  /oracle/CSIFUSION/instance/CommonDomain_webtier/diagnostics/logs/OHS/ohs1/console~OHS~1.log



Log file shows:

[2016-11-01T19:22:27.8269+05:30] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10283] [tid: 46980419799328] [user: oracle] [VirtualHost: main]  Unable to open logs

[2016-11-01T19:22:29.2839+05:30] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10291] [tid: 47345508682016] [user: oracle] [VirtualHost: main] (98)Address already in use:  make_sock: could not bind to address 192.168.0.200:15639

[2016-11-01T19:22:29.2840+05:30] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10291] [tid: 47345508682016] [user: oracle] [VirtualHost: main]  no listening sockets available, shutting down

[2016-11-01T19:22:29.2840+05:30] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10291] [tid: 47345508682016] [user: oracle] [VirtualHost: main]  Unable to open logs



Fix:
[oracle@fusionapp bin]$ lsof -i:15639
COMMAND   PID   USER   FD   TYPE  DEVICE SIZE NODE NAME
java    19404 oracle 1550u  IPv6 2258641       TCP fusionapp.chainsys.com:15639->fusionapp.chainsys.com:rap-listen (ESTABLISHED)
oracle  30651 oracle   18u  IPv6 2258642       TCP fusionapp.chainsys.com:rap-listen->fusionapp.chainsys.com:15639 (ESTABLISHED)
[oracle@fusionapp bin]$ kill -9 19404 30561
-bash: kill: (30561) - No such process


[oracle@fusionapp bin]$ lsof -i:15639
[oracle@fusionapp bin]$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
[oracle@fusionapp bin]$ ./opmnctl status -l

Processes in Instance: CommonDomain_webtier
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ohs1                             | OHS                |   11681 | Alive    |  568205326 |  2775788 |   0:00:12 | http:15637,https:15638,http:15625,https:15626,https:15630,http:15627,https:15628,http:15631,https:15632,http:15639,https:15640,http:15633,https:15634,http:15623,https:15624,http:15635,https:15636,http:15641,https:15642,https:12044,https:15602,http:15601

Error while starting CommonDomain_webtier in fusion applications

This error may happen if there are some residual processes running on accidental failure of the services.

Symptoms

[oracle@fusionapp bin]$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
================================================================================
opmn id=fusionapp.chainsys.com:12043
Response: 0 of 1 processes started.

ias-instance id=CommonDomain_webtier
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
  ohs1/OHS/OHS/

Error
--> Process (index=1,uid=568205325,pid=10291)
  failed to start a managed process after the maximum retry limit
  Log:
  /oracle/CSIFUSION/instance/CommonDomain_webtier/diagnostics/logs/OHS/ohs1/console~OHS~1.log



Log file shows:

[2016-11-01T19:22:27.8269+05:30] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10283] [tid: 46980419799328] [user: oracle] [VirtualHost: main]  Unable to open logs

[2016-11-01T19:22:29.2839+05:30] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10291] [tid: 47345508682016] [user: oracle] [VirtualHost: main] (98)Address already in use:  make_sock: could not bind to address 192.168.0.200:15639

[2016-11-01T19:22:29.2840+05:30] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10291] [tid: 47345508682016] [user: oracle] [VirtualHost: main]  no listening sockets available, shutting down

[2016-11-01T19:22:29.2840+05:30] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: fusionapp.chainsys.com] [host_addr: 192.168.0.200] [pid: 10291] [tid: 47345508682016] [user: oracle] [VirtualHost: main]  Unable to open logs



Fix:
[oracle@fusionapp bin]$ lsof -i:15639
COMMAND   PID   USER   FD   TYPE  DEVICE SIZE NODE NAME
java    19404 oracle 1550u  IPv6 2258641       TCP fusionapp.chainsys.com:15639->fusionapp.chainsys.com:rap-listen (ESTABLISHED)
oracle  30651 oracle   18u  IPv6 2258642       TCP fusionapp.chainsys.com:rap-listen->fusionapp.chainsys.com:15639 (ESTABLISHED)
[oracle@fusionapp bin]$ kill -9 19404 30561
-bash: kill: (30561) - No such process


[oracle@fusionapp bin]$ lsof -i:15639
[oracle@fusionapp bin]$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
[oracle@fusionapp bin]$ ./opmnctl status -l

Processes in Instance: CommonDomain_webtier
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ohs1                             | OHS                |   11681 | Alive    |  568205326 |  2775788 |   0:00:12 | http:15637,https:15638,http:15625,https:15626,https:15630,http:15627,https:15628,http:15631,https:15632,http:15639,https:15640,http:15633,https:15634,http:15623,https:15624,http:15635,https:15636,http:15641,https:15642,https:12044,https:15602,http:15601