Get help with morph.io and scraping

Node.js scrapers failing

I have multiple scrapers failing with

Error: Invalid semantic version "14.x"

for 3 days now.

2 Likes

The problem still exists. I can’t find any project with node.js (for copying the settings) which ran successfully in the last few months.
It seems that there is something wrong in the library, in heroku. This semver spec seems quite clear, but I already tried strings like “12.x”, “16.13.0”, “8.1.0”, “14.x”, “6.x” for the engines.node and engines.npm setting in the package.json. Before this problem, I hadn’t this setting at all in my file.

package.json:

  "engines": {
    "node": "12.x"
  },

Error message:

Injecting configuration and compiling...
e[1G       e[1G-----> Node.js app detected
e[1G       
e[1G-----> Creating runtime environment
e[1G       
e[1G       NPM_CONFIG_LOGLEVEL=error
e[1G       NPM_CONFIG_PRODUCTION=true
e[1G       NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt
e[1G       NODE_VERBOSE=false
e[1G       NODE_ENV=production
e[1G       NODE_TLS_REJECT_UNAUTHORIZED=0
e[1G       NODE_MODULES_CACHE=true
e[1G       
e[1G-----> Installing binaries
e[1G       engines.node (package.json):  12.x
e[1G       engines.npm (package.json):   unspecified (use default)
e[1G       
e[1G       Resolving node version 12.x...
e[1G       Error: Invalid semantic version "12.x"
e[1G       
e[1G-----> Build failed
e[1G       
e[1G       !     Invalid semver requirement
e[1G       
e[1G       Node, Yarn, and npm adhere to semver, the semantic versioning convention
e[1G       popularized by GitHub.
e[1G       
e[1G       http://semver.org/
e[1G       
e[1G       However you have specified a version requirement that is not a valid
e[1G       semantic version.
e[1G       
e[1G       https://kb.heroku.com/why-is-my-node-js-build-failing-because-of-an-invalid-semver-requirement
e[1G       

I even tried creating a new project to see what is the new default, but creating new project hangs with “Add scraper template …”. And even the values from the documentation example are not working:
https://morph.io/documentation/language_version

Update: it even fails with the exact example values from heroku: Why is my Node.js build failing because of an invalid semver requirement? - Heroku Help

Workaround: let morph.io think it’s a python project:

  • Create an empty requirements.txt
  • create runtime.txt with content: “python-3.6.2”
  • rename package.json to package_orig.json (this was the indicator for a node project)
  • rename scraper.js to scraper_orig.js (maybe not necessary)
  • create a scraper.py with the following content:
import os

os.system('wget --progress=dot:mega https://nodejs.org/dist/v16.13.1/node-v16.13.1-linux-x64.tar.xz')
os.system('tar xJf node-v16.13.1-linux-x64.tar.xz')
print('wrapper: downloading done.')
os.environ['PATH'] = os.getcwd() + '/node-v16.13.1-linux-x64/bin/:' + os.environ['PATH']
os.rename('package_orig.json', 'package.json')
os.system('npm install')
print('wrapper: installing done.')
os.system('node scraper_orig.js')
print('wrapper: all done.')

This probably makes the process slower, and I don’t like to trick morph.io this way. But it is the only way to fix the issue which exists since 2 months now.
With this change, your nodejs project starts as python project, but installs nodejs and all dependencies in the temporary (probably) docker environment and runs it. be careful when testing at home.