Infinitely "queued to run"


#1

Is anyone else having trouble running a scraper on Morph? I thought it was an issue with my code, but I’ve tried to run a few scrapers today and they all just infinitely say “queued to run” without showing anything in the console output.


#2

Yeah I’ve got some stuck that have been queued for more than a day as well:

https://morph.io/wdiv-scrapers/DC-PollingStations-Hastings
https://morph.io/wdiv-scrapers/DC-PollingStations-Camden
https://morph.io/wdiv-scrapers/DC-PollingStations-Runnymede
https://morph.io/wdiv-scrapers/DC-PollingStations-Doncaster
https://morph.io/wdiv-scrapers/DC-PollingStations-Eden

Looking at the admin, it seems like there are some containers with no active and retrying job that require cleanup


#3

Thanks for checking this out @chris48s!


#4

Some of these seem to be cleared now, but there appear to be a few left behind
e.g:
https://morph.io/wdiv-scrapers/DC-PollingStations-Eden

@walinchus - are yours able to complete now?


#5

@henare @equivalentideas - any chance of getting these 2

https://morph.io/wdiv-scrapers/DC-PollingStations-Eden
https://morph.io/chris48s/OS-AddressBase-Releases-Scraper

fixed? I’m guessing there’s going to be some others that are lost somewhere in the ether…

Thanks


#6

Just getting to this now @chris48s sorry for the slow response.


#7

Following the pattern here https://github.com/openaustralia/morph/issues/910#issuecomment-141902636

2.3.1 :001 > Run.where(finished_at: nil, started_at: nil).where('queued_at IS NOT NULL').count
 => 90

The scrapers you pointed to above were in that state @chris48s

I ran:

2.3.1 :009 > Run.where(finished_at: nil, started_at: nil).where('queued_at IS NOT NULL').each { |r| RunWorker.perform_async(r.id) }

That’s kicking them off https://morph.io/chris48s/OS-AddressBase-Releases-Scraper

Thanks for letting us know about this, and sorry again it took so long for the fix!