Can anyone point me at a doc explaining the first steps of diagnosing why a scraper that works locally seems to fail when deployed and run on morph.io? So far I’ve dumped module versions along with python versions and made sure that my console output is going to stdout to be captured on the page - still no closer to understanding why. This is specifically related to the python/Phantom Issues in Scraping Javascript Heavy Sites - PhantomJS version [With Python and Selenium] and relates to project https://github.com/ianibo/SirsiDynixIBistroScraper but I figured maybe there is a useful stand-alone resource worth creating - anyone have any suggestions? ty, Ian.
Have installed ruby [via rvm] and the morph gem . Trying to run the scraper reports
ibbo@xuss:~/dev/SirsiDynixIBistroScraper$ morph
Uploading and running…
/home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/openssl/buffering.rb:326:in syswrite': Connection reset by peer (Errno::ECONNRESET) from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/openssl/buffering.rb:326:indo_write’
from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/openssl/buffering.rb:344:in write' from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/protocol.rb:211:inwrite0’
from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/protocol.rb:185:in block in write' from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/protocol.rb:202:inwriting’
from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/protocol.rb:184:in write' from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http/generic_request.rb:187:insend_request_with_body’
from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http/generic_request.rb:120:in exec' from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1412:inblock in transport_request’
from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1411:in catch' from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1411:intransport_request’
from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:1384:in request' from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/rest-client-1.8.0/lib/restclient/request.rb:270:innet_http_do_request’
from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/rest-client-1.8.0/lib/restclient/request.rb:415:in block in transmit' from /home/ibbo/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/net/http.rb:853:instart’
from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in transmit' from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/rest-client-1.8.0/lib/restclient/request.rb:176:inexecute’
from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in execute' from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/morph-cli-0.2.2/lib/morph-cli.rb:26:inexecute’
from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/morph-cli-0.2.2/bin/morph:32:in execute' from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/command.rb:27:inrun’
from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in invoke_command' from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor.rb:359:indispatch’
from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/thor-0.19.1/lib/thor/base.rb:440:in start' from /home/ibbo/.rvm/gems/ruby-2.2.1/gems/morph-cli-0.2.2/bin/morph:63:in<top (required)>'
from /home/ibbo/.rvm/gems/ruby-2.2.1/bin/morph:23:in load' from /home/ibbo/.rvm/gems/ruby-2.2.1/bin/morph:23:in'
from /home/ibbo/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in eval' from /home/ibbo/.rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in’
Hopefully it’s very rare that you encounter a difference between local runs and those on morph.io. As @otherchirps says over on the other topic it looks like you hit quite the corner case!
Hmm, that definitely shouldn’t happen. Connection reset by peer may have been a temporary problem, could you try again?
Another random possibility is that the latest version of Ruby I’ve used it on is 2.2.0. Surely that can’t be the problem but could you also try running it on 2.2.0?