Using python 3 with morph scraperwiki fork

Question & Answer
1

I’m having problems getting python 3 scrapers to save to the database on morph.io.

I set my scraper runtime to python-3.4.3, then install the morph.io fork of the scraperwiki library with this:

-e git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki

…but when I run, this fork complains about the missing urllib2 (which no longer exists in python 3.)

The latest version of the scraperwiki library on pypi is python 3 compatible, but doesn’t seem to be morph.io compatible (it saves to scraperwiki.sqlite, whereas I think morph.io requires it to save to data.sqlite.)

Any help greatly appreciated. I’d much prefer to use python 3.

1 Like
2

You’re right - all the fork does is change the database filename.

The excellent @otherchirps has a scraper that gets around this: https://github.com/otherchirps/nsw_gov_docs/blob/master/scraper.py#L4-L7

Let us know if that works for you too.

3

Great, yes – that’s fixed it. Thanks, @henare and @otherchirps

1 Like
4

I ran into this again, so I’ve opened a PR.