OpenSSL Error when trying to hit domain with Mechanize: Hostname does not match certificated

Question & Answer
1

Hi morphers,

I’ve written a little scraper to get information off the front end of policies on They Vote For You https://morph.io/equivalentideas/theyvoteforyou_policy_positions .

When Mechanize goes to get a page for the second time, it errors with:

/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/openssl/ssl.rb:232:in `post_connection_check': hostname "theyvoteforyou.org.au" does not match the server certificate (OpenSSL::SSL::SSLError)

The They Vote For You certificate has the common name ‘www.theyvoteforyou.org.au’.

They Vote For You always redirects to https.theyvoteforyou.org.au, so shouldn’t the cert not have the www? Is that a problem with the cert that needs to be corrected, or should I try and make the request without ssl? Any other more informed ideas would be very helpful :wink:

Luke

2

This scraper works for me if I run it locally so I’d say something is up with morph.io.

I created a test scraper and it worked when hitting TVFY once but then I added a second request and it started failing so something weird is going on.

https://morph.io/morph-test-scrapers/mechanize_tvfy_test

3

I’m with @henare here. As it works locally and only fails on morph.io looks like it’s probably an issue with morph.io’s transparent web proxy. I’ve created an issue on github for it: https://github.com/openaustralia/morph/issues/880

I’ll take a look at this today

4

@henare @equivalentideas did you just get an email notifying you that I mentioned you?

5

@mlandauer yes, nice one!

6

Me too!

Thanks for picking this up gang :+1: