Getting 502 Bad Gateway error using Mechanize on a scraper

Hi there,

Just recently started playing around with Morph again as QuickCode is shutting down. I’ve had a scraper working for quite a while on QuickCode (Scraperwiki) scraping course registration data. It’s working fine in QuickCode/Scraperwiki but on I get a 502 Bad Gateway Error:

Traceback (most recent call last):
File “”, line 47, in
response =, timeout=60)
File “/app/.heroku/python/lib/python2.7/site-packages/mechanize/”, line 203, in open
return self._mech_open(url, data, timeout=timeout)
File “/app/.heroku/python/lib/python2.7/site-packages/mechanize/”, line 255, in _mech_open
raise response
mechanize._response.httperror_seek_wrapper: HTTP Error 502: Bad Gateway

On Twitter, someone from said it looked like a bad certificate. And, indeed, when I go to this site in Chrome I get a warning that says the certificate is invalid, so that sounds right.

But I don’t really care about the security vulnerability here. I just want to scrape the data. Is there some way I could just get mechanize to ignore the security certificate and plow on anyway?

Here’s the scraper:



Think I figured it out: The URL of the university changed from to

The old course site was still live but that broke the certificate because now the security certificate was coming from

When I changed the URL in the scraper to (the new course site), it started working!


1 Like

At the time, the answer was “Not easily; we funnel web traffic through a proxy which validates the certificates and you can’t override that”.

However, because of this issue (and a few other things that came up at the same time) we’ve disabled that proxy, so the answer now is “Yes, it should be fairly easy”. It looks like PEP 476 – Enabling certificate verification by default for stdlib http clients | is still relevant, I think, but I haven’t tested this.

1 Like