How can I compensate for an apparent SSL chain mismatch in Jsoup?
up vote
0
down vote
favorite
I'm requesting a webpage with the following code:
Jsoup.connect(url)
.method(Connection.Method.GET)
.execute()
The site I'm attempting to scrape shows the following certification path in Chrome:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) DigiCert Global CA G2
Issued by DigiCert Global Root G2
3) DigiCert Global Root G2
Issued by DigiCert Global Root G2
Running my program with -Djavax.net.debug=ssl shows this chain:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) Symantec Class 3 Secure Server SHA256 SSL CA
Issued by VeriSign Universal Root Certification Authority
3) VeriSign Universal Root Certification Authority
Issued by VeriSign Universal Root Certification Authority
The result is a javax.net.ssl.SSLHandshakeException. Testing the server in SSL Labs shows the correct chain under "Certification Paths" and the broken one under "Additional Certificates (if supplied)", with "Incomplete, Extra certs, Contains anchor" listed as chain issues. I have no desire to use validateTLSCertificates(false) unless absolutely necessary.
How can I ensure that Jsoup follows the correct certification path instead of getting distracted by the erroneous extra certificate?
ssl ssl-certificate jsoup
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
asked Nov 8 at 19:56
Shay Guy
4661512
add a comment |
up vote
0
down vote
favorite
I'm requesting a webpage with the following code:
Jsoup.connect(url)
.method(Connection.Method.GET)
.execute()
The site I'm attempting to scrape shows the following certification path in Chrome:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) DigiCert Global CA G2
Issued by DigiCert Global Root G2
3) DigiCert Global Root G2
Issued by DigiCert Global Root G2
Running my program with -Djavax.net.debug=ssl shows this chain:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) Symantec Class 3 Secure Server SHA256 SSL CA
Issued by VeriSign Universal Root Certification Authority
3) VeriSign Universal Root Certification Authority
Issued by VeriSign Universal Root Certification Authority
The result is a javax.net.ssl.SSLHandshakeException. Testing the server in SSL Labs shows the correct chain under "Certification Paths" and the broken one under "Additional Certificates (if supplied)", with "Incomplete, Extra certs, Contains anchor" listed as chain issues. I have no desire to use validateTLSCertificates(false) unless absolutely necessary.
How can I ensure that Jsoup follows the correct certification path instead of getting distracted by the erroneous extra certificate?
ssl ssl-certificate jsoup
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
asked Nov 8 at 19:56
Shay Guy
4661512
Make sure you are using SNI. Nowadays many TLS servers, specially HTTPS one, sit on one IP but multiple names, so the certificates (and the whole TLS exchange) that the server will send back to client will depend on which hostname the client requested, as forwarded in the SNI extension.
– Patrick Mevzek
Nov 8 at 21:06
@PatrickMevzek Is SNI automatic with JVM 1.8 and Jsoup 1.11.3?
– Shay Guy
Nov 8 at 21:40
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm requesting a webpage with the following code:
Jsoup.connect(url)
.method(Connection.Method.GET)
.execute()
The site I'm attempting to scrape shows the following certification path in Chrome:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) DigiCert Global CA G2
Issued by DigiCert Global Root G2
3) DigiCert Global Root G2
Issued by DigiCert Global Root G2
Running my program with -Djavax.net.debug=ssl shows this chain:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) Symantec Class 3 Secure Server SHA256 SSL CA
Issued by VeriSign Universal Root Certification Authority
3) VeriSign Universal Root Certification Authority
Issued by VeriSign Universal Root Certification Authority
The result is a javax.net.ssl.SSLHandshakeException. Testing the server in SSL Labs shows the correct chain under "Certification Paths" and the broken one under "Additional Certificates (if supplied)", with "Incomplete, Extra certs, Contains anchor" listed as chain issues. I have no desire to use validateTLSCertificates(false) unless absolutely necessary.
How can I ensure that Jsoup follows the correct certification path instead of getting distracted by the erroneous extra certificate?
ssl ssl-certificate jsoup
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
asked Nov 8 at 19:56
Shay Guy
4661512
I'm requesting a webpage with the following code:
Jsoup.connect(url)
.method(Connection.Method.GET)
.execute()
The site I'm attempting to scrape shows the following certification path in Chrome:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) DigiCert Global CA G2
Issued by DigiCert Global Root G2
3) DigiCert Global Root G2
Issued by DigiCert Global Root G2
Running my program with -Djavax.net.debug=ssl shows this chain:
1) ***REDACTED***
Issued by DigiCert Global CA G2
2) Symantec Class 3 Secure Server SHA256 SSL CA
Issued by VeriSign Universal Root Certification Authority
3) VeriSign Universal Root Certification Authority
Issued by VeriSign Universal Root Certification Authority
The result is a javax.net.ssl.SSLHandshakeException. Testing the server in SSL Labs shows the correct chain under "Certification Paths" and the broken one under "Additional Certificates (if supplied)", with "Incomplete, Extra certs, Contains anchor" listed as chain issues. I have no desire to use validateTLSCertificates(false) unless absolutely necessary.
How can I ensure that Jsoup follows the correct certification path instead of getting distracted by the erroneous extra certificate?
ssl ssl-certificate jsoup
ssl ssl-certificate jsoup
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
asked Nov 8 at 19:56
Shay Guy
4661512
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
asked Nov 8 at 19:56
Shay Guy
4661512
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
edited Nov 12 at 8:40
Jayson Minard
35.7k13103170
35.7k13103170
asked Nov 8 at 19:56
Shay Guy
4661512
asked Nov 8 at 19:56
Shay Guy
4661512
asked Nov 8 at 19:56
Shay Guy
4661512
4661512
Make sure you are using SNI. Nowadays many TLS servers, specially HTTPS one, sit on one IP but multiple names, so the certificates (and the whole TLS exchange) that the server will send back to client will depend on which hostname the client requested, as forwarded in the SNI extension.
– Patrick Mevzek
Nov 8 at 21:06
@PatrickMevzek Is SNI automatic with JVM 1.8 and Jsoup 1.11.3?
– Shay Guy
Nov 8 at 21:40
add a comment |
Make sure you are using SNI. Nowadays many TLS servers, specially HTTPS one, sit on one IP but multiple names, so the certificates (and the whole TLS exchange) that the server will send back to client will depend on which hostname the client requested, as forwarded in the SNI extension.
– Patrick Mevzek
Nov 8 at 21:06
@PatrickMevzek Is SNI automatic with JVM 1.8 and Jsoup 1.11.3?
– Shay Guy
Nov 8 at 21:40
Make sure you are using SNI. Nowadays many TLS servers, specially HTTPS one, sit on one IP but multiple names, so the certificates (and the whole TLS exchange) that the server will send back to client will depend on which hostname the client requested, as forwarded in the SNI extension.
– Patrick Mevzek
Nov 8 at 21:06
Make sure you are using SNI. Nowadays many TLS servers, specially HTTPS one, sit on one IP but multiple names, so the certificates (and the whole TLS exchange) that the server will send back to client will depend on which hostname the client requested, as forwarded in the SNI extension.
– Patrick Mevzek
Nov 8 at 21:06
@PatrickMevzek Is SNI automatic with JVM 1.8 and Jsoup 1.11.3?
– Shay Guy
Nov 8 at 21:40
@PatrickMevzek Is SNI automatic with JVM 1.8 and Jsoup 1.11.3?
– Shay Guy
Nov 8 at 21:40
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53215228%2fhow-can-i-compensate-for-an-apparent-ssl-chain-mismatch-in-jsoup%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Make sure you are using SNI. Nowadays many TLS servers, specially HTTPS one, sit on one IP but multiple names, so the certificates (and the whole TLS exchange) that the server will send back to client will depend on which hostname the client requested, as forwarded in the SNI extension.
– Patrick Mevzek
Nov 8 at 21:06
@PatrickMevzek Is SNI automatic with JVM 1.8 and Jsoup 1.11.3?
– Shay Guy
Nov 8 at 21:40