Ashoka,

I will need a little more information to get an good understanding of the current state of your server.

1) Which certificate expired? Was it all of them? (ie Host certificate and user certificates?)

2) When following the instructions at https://www.labkey.org/wiki/home/Documentation/page.view?name=globusServer did you create a new Certificate Authority? If you did not create a new certificate authority, did you renew the existing certificates or create new ones?

3) DId you create new or renew certificates for users and for the host certificate?

4) If you created a new or renewed your host certificates, did you restart the Globus server?

Can you send me a copy of your globus logs too? (send this to my email as it is probably a large file).

Thanks

Brian

Thanks Brian. See below. I'll send the log file to your email. Is it under /usr/local/gt4.0.6/var ?

1) User certificate expired. Tried to create a new one, sign and use it. Did not work. So we tried creating a new authority.
2. Did create both host and user certificates.
3. New for both host and user.
4. Yes we restated the Globus server.

-Ashoka

Ashoka,

When you see an error like

"...Bad certificate (The signature of 'O=Grid,OU=GlobusTest,OU=simpleCA-medusa.tgen.org,CN=host/medusa.tgen.org' certificate does not match its issuer).."

This usually means that the host certificate (/etc/grid-security/hostcert.pem) or container certificate (/etc/grid-security/containercert.pem) were issued by a different certificate authority than the one currently in use.

My guess is that you are currently running in one of two states

1) host certificate and/or container certificate is from old certificate authority and new certificate authority is configured as default.

2) host certificate and/or container certificate is from new certificate authority and old certificate authority is configured as default.


When the new certificate authority was created, it should have created some new files in the /etc/grid-security/certificates directory. One of these files is the root certificate for the new CA. It should be of the form xxxxxxxx.0 where "xxxxxxxx" is an alphanumeric string that was generated during the certificate authority creation.

Does this file exist? If so, it is read-able by the globus user?

You can test if the current host certificate was signed by the a given certificate authority by running

   openssl verify -CAfile = /etc/grid-security/certificates/xxxxxxxx.0 /etc/grid-security/hostcert.pem

-where xxxxxxxx.0 is the root certificate for your CA.

If certificate was created by the certificate authority, you will see a message like
"hostcert.pem: OK"

You should test the root cert from both the old and the new certificate authority. In addition, you should run the same test with the containercert.pem


Lastly, you can view the certificate that is currently in use by the Globus Server by running

    openssl s_client -connect medusa.tgen.org:8443

-this command will output the entire certificate chain currently in use by the globus server. You can use this output to determine if this certificate was created by the old or new certificate authority.

(if you use the command "openssl x509 -in hostcert.pem -text -noout", it will output the certificate in text format. You can use this to compare to output of command above. )


What you want to ensure is that for the hostcert.pem/containercert.pem and user certificates you are using, the root certificate (ie one named xxxxxxxx.0) for the certificate authority which created these certificates is located in /etc/grid-security/certificates

-Brian

Hey Brian,
Both hostcert.pem and containercert.pem passed the test and displayed the messages like "hostcert.pem: OK".

Running openssl s_client -connect medusa.tgen.org:8443 returns:
CONNECTED(00000003)
depth=0 /O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
verify error:num=27:certificate not trusted
verify return:1
depth=0 /O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
   i:/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=Globus Simple CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICUzCCAbygAwIBAgIBATANBgkqhkiG9w0BAQQFADBiMQ0wCwYDVQQKEwRHcmlk
MRMwEQYDVQQLEwpHbG9idXNUZXN0MSEwHwYDVQQLExhzaW1wbGVDQS1tZWR1c2Eu
dGdlbi5vcmcxGTAXBgNVBAMTEEdsb2J1cyBTaW1wbGUgQ0EwHhcNMTEwOTI5MjIw
NzE5WhcNMTIwOTI4MjIwNzE5WjBmMQ0wCwYDVQQKEwRHcmlkMRMwEQYDVQQLEwpH
bG9idXNUZXN0MSEwHwYDVQQLExhzaW1wbGVDQS1tZWR1c2EudGdlbi5vcmcxHTAb
BgNVBAMTFGhvc3QvbWVkdXNhLnRnZW4ub3JnMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDriVfoRA7z8MxQ3FinxVr9FXQSItAvi791PtQROUQu7KHnkaLc2dvr
veQeweQTRBvo7ZurTECgJY62XwYla+8oUFy0UxUJBMyTRYNnN1xacoSuQd+yfRMu
6qFtJj+5aWlFZUsYuWQSo0isFzTGcFs3ARC5VawjoUCJaVv9LLv7VQIDAQABoxUw
EzARBglghkgBhvhCAQEEBAMCBPAwDQYJKoZIhvcNAQEEBQADgYEAVomYIE5owLdw
StAYFZfp3QOGPGgBy11qhx6UQvx5rwSyKtxwP/Ljw7LfqCmk9XwYfz5VJO5A9dMz
0c92SwOD+ZiqT4J8vT9Zs3ZPQ7CidcaIORvx43BUbj/0VG+jipBIC/vjfWjlgRLb
LhCiOmZ8+EtWH0xKTdoD+MO6jfzClkw=
-----END CERTIFICATE-----
subject=/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
issuer=/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=Globus Simple CA
---
Acceptable client certificate CA names
/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=Globus Simple CA
---
SSL handshake has read 893 bytes and written 337 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol : SSLv3
    Cipher : DES-CBC3-SHA
    Session-ID: 8F3B2D96FF92BB1D041970598CF28B5635A8B9098FACA9D3B8C1E3DE6CC1235B
    Session-ID-ctx:
    Master-Key: A025B7752A5AA1478004CDCF9D081EAC9AFAFC6E32F95642B4898524BA2542D57B29452B6F072D7C8D9725AA530B061A
    Key-Arg : None
    Krb5 Principal: None
    Start Time: 1317335367
    Timeout : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

Running openssl x509 -in /etc/grid-security/hostcert.pem -text -noout returns:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: md5WithRSAEncryption
        Issuer: O=Grid, OU=GlobusTest, OU=simpleCA-medusa.tgen.org, CN=Globus Simple CA
        Validity
            Not Before: Sep 29 22:07:19 2011 GMT
            Not After : Sep 28 22:07:19 2012 GMT
        Subject: O=Grid, OU=GlobusTest, OU=simpleCA-medusa.tgen.org, CN=host/medusa.tgen.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (1024 bit)
                Modulus (1024 bit):
                    00:eb:89:57:e8:44:0e:f3:f0:cc:50:dc:58:a7:c5:
                    5a:fd:15:74:12:22:d0:2f:8b:bf:75:3e:d4:11:39:
                    44:2e:ec:a1:e7:91:a2:dc:d9:db:eb:bd:e4:1e:c1:
                    e4:13:44:1b:e8:ed:9b:ab:4c:40:a0:25:8e:b6:5f:
                    06:25:6b:ef:28:50:5c:b4:53:15:09:04:cc:93:45:
                    83:67:37:5c:5a:72:84:ae:41:df:b2:7d:13:2e:ea:
                    a1:6d:26:3f:b9:69:69:45:65:4b:18:b9:64:12:a3:
                    48:ac:17:34:c6:70:5b:37:01:10:b9:55:ac:23:a1:
                    40:89:69:5b:fd:2c:bb:fb:55
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Cert Type:
                SSL Client, SSL Server, S/MIME, Object Signing
    Signature Algorithm: md5WithRSAEncryption
        56:89:98:20:4e:68:c0:b7:70:4a:d0:18:15:97:e9:dd:03:86:
        3c:68:01:cb:5d:6a:87:1e:94:42:fc:79:af:04:b2:2a:dc:70:
        3f:f2:e3:c3:b2:df:a8:29:a4:f5:7c:18:7f:3e:55:24:ee:40:
        f5:d3:33:d1:cf:76:4b:03:83:f9:98:aa:4f:82:7c:bd:3f:59:
        b3:76:4f:43:b0:a2:75:c6:88:39:1b:f1:e3:70:54:6e:3f:f4:
        54:6f:a3:8a:90:48:0b:fb:e3:7d:68:e5:81:12:db:2e:10:a2:
        3a:66:7c:f8:4b:56:1f:4c:4a:4d:da:03:f8:c3:ba:8d:fc:c2:
        96:4c

Any ideas how to compare?
-Ashoka

Ashoka,

You will want to compare the chunk of text between the BEGIN CERTIFICATE and END CERTIFICATE lines in your previous email message against the contents of the hostcert.pem and the containercert.pem files. A simple diff should tell you if they are the same.

If these are the same, another possibility is that private key does not match the certificate. For both the hostcert and containercert verify that the associated private key (ie hostkey.pem) is correct, follow the instructions in the section "Verify A Certificate Matches A Private Key " at http://security.ncsa.illinois.edu/research/grid-howtos/usefulopenssl.html

If private keys and certificates are correct, then I would like you follow the instructions in the "Test the WS-GRAM installation" section on https://www.labkey.org/wiki/home/Documentation/page.view?name=globusServer for instructions on testing your implementation. This may help us isolate where in the globus server the problem exists.

-Brian

Ashoka,

You will want to compare the chunk of text between the BEGIN CERTIFICATE and END CERTIFICATE lines in your previous email message against the contents of the hostcert.pem and the containercert.pem files. A simple diff should tell you if they are the same.

If these are the same, another possibility is that private key does not match the certificate. For both the hostcert and containercert verify that the associated private key (ie hostkey.pem) is correct, follow the instructions in the section "Verify A Certificate Matches A Private Key " at http://security.ncsa.illinois.edu/research/grid-howtos/usefulopenssl.html

If private keys and certificates are correct, then I would like you follow the instructions in the "Test the WS-GRAM installation" section on https://www.labkey.org/wiki/home/Documentation/page.view?name=globusServer for instructions on testing your implementation. This may help us isolate where in the globus server the problem exists.

-Brian

Brian,
I am taking a look at this issue and get the following errors when running the openssl command, however I did run all the tests in the WS-GRAM testing and it was successful with all of those. Note the results of the openssl command are different from the results Ashoka posted above.. I am not sure if there were any changes to the server since those original results were posted. Please let me know how we should proceed.

Thanks,
James Lowey



[labkey@medusa globus_test]$ openssl s_client -connect medusa.tgen.org:8443
CONNECTED(00000003)
depth=0 /O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
verify error:num=27:certificate not trusted
verify return:1
depth=0 /O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
   i:/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=Globus Simple CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICUzCCAbygAwIBAgIBATANBgkqhkiG9w0BAQQFADBiMQ0wCwYDVQQKEwRHcmlk
MRMwEQYDVQQLEwpHbG9idXNUZXN0MSEwHwYDVQQLExhzaW1wbGVDQS1tZWR1c2Eu
dGdlbi5vcmcxGTAXBgNVBAMTEEdsb2J1cyBTaW1wbGUgQ0EwHhcNMTEwOTI5MjIw
NzE5WhcNMTIwOTI4MjIwNzE5WjBmMQ0wCwYDVQQKEwRHcmlkMRMwEQYDVQQLEwpH
bG9idXNUZXN0MSEwHwYDVQQLExhzaW1wbGVDQS1tZWR1c2EudGdlbi5vcmcxHTAb
BgNVBAMTFGhvc3QvbWVkdXNhLnRnZW4ub3JnMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDriVfoRA7z8MxQ3FinxVr9FXQSItAvi791PtQROUQu7KHnkaLc2dvr
veQeweQTRBvo7ZurTECgJY62XwYla+8oUFy0UxUJBMyTRYNnN1xacoSuQd+yfRMu
6qFtJj+5aWlFZUsYuWQSo0isFzTGcFs3ARC5VawjoUCJaVv9LLv7VQIDAQABoxUw
EzARBglghkgBhvhCAQEEBAMCBPAwDQYJKoZIhvcNAQEEBQADgYEAVomYIE5owLdw
StAYFZfp3QOGPGgBy11qhx6UQvx5rwSyKtxwP/Ljw7LfqCmk9XwYfz5VJO5A9dMz
0c92SwOD+ZiqT4J8vT9Zs3ZPQ7CidcaIORvx43BUbj/0VG+jipBIC/vjfWjlgRLb
LhCiOmZ8+EtWH0xKTdoD+MO6jfzClkw=
-----END CERTIFICATE-----
subject=/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=host/medusa.tgen.org
issuer=/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=Globus Simple CA
---
Acceptable client certificate CA names
/O=Grid/OU=GlobusTest/OU=simpleCA-medusa.tgen.org/CN=Globus Simple CA
---
SSL handshake has read 893 bytes and written 337 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol : SSLv3
    Cipher : DES-CBC3-SHA
    Session-ID: 1E64BEFE629633BE91AA6C9256DE0E80780859A3F4BEF2B01E77CB7CF6B3A50C
    Session-ID-ctx:
    Master-Key: DAD653B0CFD5ED8229D868613C1EE951140F52A85E8332D9B21540A674DFFD2410CDDCB7A1202D1945D9 B08898ADBF70
    Key-Arg : None
    Krb5 Principal: None
    Start Time: 1319651400
    Timeout : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

HTTP/1.1 404
Content-Type: text/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: close

3a8
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3 .org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <soapenv:Body>
  <soapenv:Fault>
   <faultcode>soapenv:Server.userException</faultcode>
   <faultstring>java.io.IOException: Unexpected end of stream</faultstring>
   <detail>
    <ns1:stackTrace xmlns:ns1="http://xml.apache.org/axis/">java.io.IOException: Unexpected end of s tream
        at org.globus.wsrf.container.ServiceThread.parseHeaders(ServiceThread.java:824)
        at org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:330)
        at org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:153)
        at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:291)
</ns1:stackTrace>
    <ns2:hostname xmlns:ns2="http://xml.apache.org/axis/">medusa</ns2:hostname>
   </detail>
  </soapenv:Fault>
 </soapenv:Body>
</soapenv:Envelope>
0

read:errno=104

James,
Given that some things might have changed since our initial testing, I recommend we jump on web conference where we can debug the configuration in real-time. This will allow us to quickly get your pipeline back up and working again.

I will send you a direct email to schedule the web conference.

Brian