As a break from issues with SqlLite not storing data, I have a different issue:
I’m working on this scraper:
It looks like it takes a little while for the calendar to load the first time and I’m guessing this is why it’s failing. You can wait for it to finish loading by applying a no-small-dose of RSpec:
@@url = "http://hansardpublic.parliament.sa.gov.au/#/search/1"
@session = Capybara::Session.new(:poltergeist)
@session.driver.browser.js_errors = false
expect(@session).to(have_css('.scheduler .yearWrapper .k-widget', wait: 10))
warn 'all set!'
CalendarSearch.new.ready do |session|
# do stuff with `session`
Or you can just sleep for a few secs at the beginning and hope for the best
What version of PhantomJS is on Morph.io?
It’s just installed using apt-get so whatever version Ubuntu has, I guess. You could always write a scraper to find out
Thankyou @wfdd - that really helps!
I will play with it next week or the week after when I get a chance to code.
It would be helpful to have scrapers running on Morph.io not just locally. Then other people can use them or reuse the code for other stuff
@henare LOL at writing a scraper to find out.
As an aside, when using apt install PhantomJS on my Ubuntu 16.* VPS, it installed great but errored when run. Something about the repo version being compiled against Unity or another graphics library. Ended up using the compiled-from-source version.
Hey, I just got a couple of lessons in proper code structuring in Ruby, from this (bonus). Thanks @wfdd
Got it working! Sweet. Thankyou for all your help!