Getting 502 Bad Gateway error using Mechanize on a scraper

chadskelton · June 25, 2019, 5:31pm

Hi there,

Just recently started playing around with Morph again as QuickCode is shutting down. I’ve had a scraper working for quite a while on QuickCode (Scraperwiki) scraping course registration data. It’s working fine in QuickCode/Scraperwiki but on Morph.io I get a 502 Bad Gateway Error:

Traceback (most recent call last):
File “scraper.py”, line 47, in
response = br.open(url, timeout=60)
File “/app/.heroku/python/lib/python2.7/site-packages/mechanize/_mechanize.py”, line 203, in open
return self._mech_open(url, data, timeout=timeout)
File “/app/.heroku/python/lib/python2.7/site-packages/mechanize/_mechanize.py”, line 255, in _mech_open
raise response
mechanize._response.httperror_seek_wrapper: HTTP Error 502: Bad Gateway

On Twitter, someone from Morph.io said it looked like a bad certificate. And, indeed, when I go to this site in Chrome I get a warning that says the certificate is invalid, so that sounds right.

But I don’t really care about the security vulnerability here. I just want to scrape the data. Is there some way I could just get mechanize to ignore the security certificate and plow on anyway?

Here’s the scraper:

github.com

chadskelton/course_scraper/blob/master/scraper.py

#!/usr/bin/env python

# NEED TO CHANGE TERM CODE FOR IT TO WORK EACH TIME 
# SEEMS TO BE DROPPING COURSES FOR REASONS THAT AREN'T CLEAR; HAS MOST JRNL COURSES BUT ONLY ONE COMM COURSE
# FIXED BY GRABBING ALL ROWS ON THE PAGE INSTEAD OF TABLE FIRST, THEN ROWS. BECAUSE CHECK FOR CHECKBOX BEFORE ADDING TO TABLE, WORKS MORE SIMPLY
# https://stackoverflow.com/questions/15603561/how-can-i-debug-a-http-post-in-chrome
# Go to View>Developer Tools; Network; Filter to method:POST; Click on it and then Headers
# Also some instructions here: http://www.exegetic.biz/blog/2016/09/viewing-post-data/

import mechanize 
import lxml.html
import time
import urllib2
import scraperwiki
from datetime import datetime
from BeautifulSoup import BeautifulSoup
import ssl

br = mechanize.Browser()
br.set_handle_robots(False)

This file has been truncated. show original

Thanks!

Chad

chadskelton · June 26, 2019, 2:13am

Think I figured it out: The URL of the university changed from kwantlen.ca to kpu.ca

The old kwantlen.ca course site was still live but that broke the certificate because now the security certificate was coming from kpu.ca

When I changed the URL in the scraper to kpu.ca (the new course site), it started working!

Chad

jamezpolley · July 9, 2019, 1:36am

At the time, the answer was “Not easily; we funnel web traffic through a proxy which validates the certificates and you can’t override that”.

However, because of this issue (and a few other things that came up at the same time) we’ve disabled that proxy, so the answer now is “Yes, it should be fairly easy”. It looks like PEP 476 – Enabling certificate verification by default for stdlib http clients | peps.python.org is still relevant, I think, but I haven’t tested this.