faststartstop utility fails with [Security:090479]Certificate chain received from fusionapp - 192.168.0.200 failed date validity checks.

Today,

when about to start fusion applications using faststartstop utility,

We encountered a problem like:

[TRACE:16] [] [WLC] [oracle.apps.startstop.util.WLSTCommand: checkNodeManagerStatus.785] [tid:10] Calling command nmConnect() in Thread#Main Thread for domain BIDomain
[2013-08-16T14:45:26.026+05:30] [fastartstop] [ERROR:1] [SSUTIL023] [Util] [oracle.apps.startstop.util.WLSTCommand: checkNodeManagerStatus.795] [tid:10] Error connecting to Node manager on host [fusionapp:5557] for domain BIDomain. Please verify node manager status.
[2013-08-16T14:45:26.026+05:30] [fastartstop] [ERROR:1] [] [Util] [oracle.apps.startstop.util.WLSTCommand: checkNodeManagerStatus.795] [tid:10] SSUTIL023
Traceback (innermost last):
  File "", line 1, in ?
  File "", line 123, in nmConnect
  File "", line 648, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : [Security:090479]Certificate chain received from fusionapp - 192.168.0.200 failed date validity checks.
Use dumpStack() to view the full stacktrace

        at org.python.core.Py.makeException(Unknown Source)
        at org.python.pycode._pyx2.raiseWLSTException$38(:648)
        at org.python.pycode._pyx2.call_function()
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.pycode._pyx2.nmConnect$3(:123)
        at org.python.pycode._pyx2.call_function()
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyFunction.__call__(Unknown Source)
        at org.python.core.PyObject.__call__(Unknown Source)
        at org.python.pycode._pyx12.f$0(:1)
        at org.python.pycode._pyx12.call_function()
        at org.python.core.PyTableCode.call(Unknown Source)
        at org.python.core.PyCode.call(Unknown Source)
        at org.python.core.Py.runCode(Unknown Source)
        at org.python.core.Py.exec(Unknown Source)
        at org.python.util.PythonInterpreter.exec(Unknown Source)
        at weblogic.management.scripting.utils.WLSTInterpreter.exec(WLSTInterpreter.java:509)
        at oracle.apps.startstop.util.WLSTCommand.checkNodeManagerStatus(WLSTCommand.java:789)
        at oracle.apps.startstop.util.SSUtils.checkNodeManagerOnAllAdminHosts(SSUtils.java:376)
        at oracle.apps.startstop.invoke.InvocationMgrImpl.invoke(InvocationMgrImpl.java:208)
        at oracle.apps.startstop.cli.FAEnv.runStartStop(FAEnv.java:109)
        at oracle.apps.startstop.cli.FAEnv.main(FAEnv.java:170)

[2013-08-16T14:45:26.026+05:30] [fastartstop] [ERROR:1] [SSUTIL023] [UTIL] [oracle.apps.startstop.util.SSUtils: checkNodeManagerOnAllAdminHosts.383] [tid:10] Error connecting to Node manager on host [fusionapp:5557] for domain BIDomain. Please verify node manager status.


FIX:

Navigate to fusionapplications weblogic home/server/lib

 cd wlserver_10.3/server/lib
[oracle@fusionapp lib]$ ls -lrt *.jks
-rwxr-x--- 1 oracle dba  2629 Jan 16  2013 DemoTrust.jks
-rwxr-x--- 1 oracle dba   994 Jan 16  2013 DemoIdentity.jks
-rw-r--r-- 1 oracle dba  1378 Jan 16  2013 fusionapp_fusion_identity.jks
-rw-r--r-- 1 oracle dba 82498 Jan 17  2013 fusion_trust.jks

[oracle@fusionapp lib]$ keytool -list -v -keystore fusionapp_fusion_identity.jks -storepass Welcome1

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: fusionapp_fusion
Creation date: Jan 16, 2013
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=fusionapp.chainsys.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Issuer: CN=fusionapp.chainsys.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Serial number: 50f6a124
Valid from: Wed Jan 16 18:16:28 IST 2013 until: Mon Jul 15 18:16:28 IST 2013
Certificate fingerprints:
         MD5:  A9:C5:1E:7A:88:1F:C1:0B:26:38:B7:C1:E9:E9:FD:15
         SHA1: F8:61:53:4D:99:CC:25:DE:2C:96:A6:A7:3D:D1:3E:4F:54:1D:26:0E
         Signature algorithm name: SHA1withRSA
         Version: 3


*******************************************
*******************************************


cp fusionapp_fusion_identity.jks fusionapp_fusion_identity.jks_bk
[oracle@fusionapp lib]$ cd /tmp
[oracle@fusionapp tmp]$ mkdir certificate
[oracle@fusionapp tmp]$ cd certificate

[oracle@fusionapp certificate]$ keytool -genkeypair -keyalg RSA -alias "fusionapp_fusion" -keystore fusionapp_fusion_identity.jks -validity 365 -dname 'CN=fusionapp.chainsys.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US'
Enter keystore password:
Re-enter new password:
Enter key password for
        (RETURN if same as keystore password):

-------------------
[oracle@fusionapp certificate]$ keytool -export -alias fusionapp_fusion -rfc -keystore fusionapp_fusion_identity.jks -file fusionapp_fusion_identity.cer
Enter keystore password:
Certificate stored in file
---------------------

[oracle@fusionapp certificate]$ ls -lrt
total 8
-rw-r--r-- 1 oracle dba 1378 Aug 16 15:22 fusionapp_fusion_identity.jks
-rw-r--r-- 1 oracle dba  865 Aug 16 15:29 fusionapp_fusion_identity.cer
[oracle@fusionapp certificate]$
----------
[oracle@fusionapp lib]$ keytool -delete -alias fusionapp_fusion -keystore fusion_trust.jks
Enter keystore password:
[oracle@fusionapp lib]$

--------------

[oracle@fusionapp lib]$ keytool -import -alias fusionapp_fusion -keystore fusion_trust.jks -file fusionapp_fusion_identity.cer
Enter keystore password:
Owner: CN=fusionapp.chainsys.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Issuer: CN=fusionapp.chainsys.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Serial number: 520df643
Valid from: Fri Aug 16 15:22:03 IST 2013 until: Sat Aug 16 15:22:03 IST 2014
Certificate fingerprints:
         MD5:  AC:47:60:B0:12:5F:0B:83:CD:32:CE:E7:07:5B:56:EF
         SHA1: 66:7E:CB:E4:26:12:0D:63:CD:C1:9B:98:2A:7E:15:DC:BC:B2:40:DA
         Signature algorithm name: SHA1withRSA
         Version: 3
Trust this certificate? [no]:  Yes
Certificate was added to keystore
[oracle@fusionapp lib]$

---------


[oracle@fusionapp lib]$ cp /tmp/certificate/fusionapp_fusion_identity.jks .
[oracle@fusionapp lib]$ keytool -list -v -keystore fusionapp_fusion_identity.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: fusionapp_fusion
Creation date: Aug 16, 2013
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=fusionapp.chainsys.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Issuer: CN=fusionapp.chainsys.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Serial number: 520df643
Valid from: Fri Aug 16 15:22:03 IST 2013 until: Sat Aug 16 15:22:03 IST 2014
Certificate fingerprints:
         MD5:  AC:47:60:B0:12:5F:0B:83:CD:32:CE:E7:07:5B:56:EF
         SHA1: 66:7E:CB:E4:26:12:0D:63:CD:C1:9B:98:2A:7E:15:DC:BC:B2:40:DA
         Signature algorithm name: SHA1withRSA
         Version: 3


*******************************************
*******************************************


Now start the nodemanager, managed servers.

Now it should start normally.



Cheers!!!