Get help with morph.io and scraping

Infinitely "queued to run"

Is anyone else having trouble running a scraper on Morph? I thought it was an issue with my code, but I’ve tried to run a few scrapers today and they all just infinitely say “queued to run” without showing anything in the console output.

Yeah I’ve got some stuck that have been queued for more than a day as well:

https://morph.io/wdiv-scrapers/DC-PollingStations-Hastings
https://morph.io/wdiv-scrapers/DC-PollingStations-Camden
https://morph.io/wdiv-scrapers/DC-PollingStations-Runnymede
https://morph.io/wdiv-scrapers/DC-PollingStations-Doncaster
https://morph.io/wdiv-scrapers/DC-PollingStations-Eden

Looking at the admin, it seems like there are some containers with no active and retrying job that require cleanup

1 Like

Thanks for checking this out @chris48s!

1 Like

Some of these seem to be cleared now, but there appear to be a few left behind
e.g:
https://morph.io/wdiv-scrapers/DC-PollingStations-Eden

@walinchus - are yours able to complete now?

1 Like

@henare @equivalentideas - any chance of getting these 2

https://morph.io/wdiv-scrapers/DC-PollingStations-Eden
https://morph.io/chris48s/OS-AddressBase-Releases-Scraper

fixed? I’m guessing there’s going to be some others that are lost somewhere in the ether…

Thanks

Just getting to this now @chris48s sorry for the slow response.

Following the pattern here https://github.com/openaustralia/morph/issues/910#issuecomment-141902636

2.3.1 :001 > Run.where(finished_at: nil, started_at: nil).where('queued_at IS NOT NULL').count
 => 90

The scrapers you pointed to above were in that state @chris48s

I ran:

2.3.1 :009 > Run.where(finished_at: nil, started_at: nil).where('queued_at IS NOT NULL').each { |r| RunWorker.perform_async(r.id) }

That’s kicking them off https://morph.io/chris48s/OS-AddressBase-Releases-Scraper

Thanks for letting us know about this, and sorry again it took so long for the fix!