I’m having trouble figuring out which version of python I can use when creating a new scraper. I’ve walked through the new scraper wizard, and have a template python scraper in my repository. I changed the python version in runtime.txt to 3.11.2, which is what I happened to have on my debian bookworm desktop, submitted it with the morph cli, and got an error about incompatible version.
; morph
Uploading 1.50 KiB...
Injecting configuration and compiling...
-----> Python app detected
! The latest version of Python 3 is python-3.6.2 (you are using python-3.11.2, which is unsupported).
! We recommend upgrading by specifying the latest version (python-3.6.2).
Learn More: https://devcenter.heroku.com/articles/python-runtimes
-----> Installing python-3.11.2
! Requested runtime (python-3.11.2) is not available for this stack (cedar-14).
! Aborting. More info: https://devcenter.heroku.com/articles/python-support
Fair enough. So I changed the python version in runtime.txt to the version suggested, unfortunately this isn’t working either.
; morph
Uploading 1.50 KiB...
Injecting configuration and compiling...
-----> Python app detected
-----> Installing python-3.6.2
! Requested runtime (python-3.6.2) is not available for this stack (cedar-14).
! Aborting. More info: https://devcenter.heroku.com/articles/python-support
I’ve also attempted this with the 2.7 python version in the template, being told I need to use 2.7.14 and then that the python-2.7.14 runtime isn’t available.
What am I doing wrong here? What do I need to do to get a working python runtime?
I think I’ve answered my own question, by eventually finding a functional python scraper and looking at what it was doing*. Here’s what’s working for me now:
Create a file named platform in the repository root, containing the word early_release. This ensures you’re running on the most recent platform morph supports, heroku-18.
Set the python version in runtime.txt to python-3.6.2
Add these required packages to requirements.txt. I know pandas and numpy are definitely required, morph will fail to start the scraper without them. I haven’t tried without the others as I intend to use them.
I’ve also found that testing by running the morph CLI doesn’t work for me, it doesn’t seem to read the platform file. You’ll have to commit and push these changes and then do a manual run in the morph UI. But at this point it should be running - the default scraper.py still fails because it doesn’t create the data.sqlite file. But it’s progress. I guess moving forward I’ll be testing my code in a venv.
OK, so I’m still having a lot of trouble with this. Trying to make any changes at all to the requirements results in not being able to find the runtime. Any changes at all. If I copy the requirements.txt from the original repo it works. If I remove the comments from that file it does not.
I did find another functioning python scraper, this one specifies heroku-18 for the platform, and has enough modules in its requirements that I can write useful code. So I’ll continue with this for now.
But I’m seriously stumped about how all of this could be so brittle.
Same here, none of my scrapers is working, due to versions. Switched and followed your advice but can’t figure out.
Is Morph.io still a running project? Any support? Did you figure out your issues?
