The queue seems to have been stuck for over a day and a half now…
Thanks @tmtmtmtm. Runs were being queued but not being run by Sidekiq.
I tried restarting the Sidekiq workers but that didn’t work. I just bounced the machine and it’s working again.
Ping @mlandauer.
Looks like there’s something awry again. None of our scrapers have run/completed in about 18 hours, and the queue is full of scrapers that have been running for multiple weeks. I’ve just manually stopped all our ones, but there seems to be something deeper wrong.
Edit: Actually it looks like some of our scrapers did run OK, but there still seems to be something up with the queue…
It looks fine to me. There’s hardly anything in the queue and manually running a scraper just now worked instantly.
Can you be more specific so I can look into please?
It might just be that https://morph.io/scrapers/running has a lot of long-running scrapers. I’m not sure if this is actually ordered by when they started, but a random sampling shows that almost everything on the list at the minute has been stalled for more than 24 hours until you get down to the bottom couple.
My guess is that me killing all our stalled entries from that has freed up more slots for everything else, so things are fine again for now, but I don’t really know how the queueing works.
That page is a bit confusing as it doesn’t really show the status of the queue. That’s what I was hoping to address in this issue but I think we’re instead working to fix the “stuck scrapers” problem altogether.
If you’ve got some stuck scrapers that need looking at can you please add them to this issue so Matthew can look into them?