My scraper stuck, what to do?

Question & Answer
82

I have a number of scrapers that appear to be stuck. e.g:
https://morph.io/wdiv-scrapers/DC-PollingStations-York
https://morph.io/wdiv-scrapers/DC-PollingStations-Shepway
https://morph.io/wdiv-scrapers/DC-PollingStations-Thanet
https://morph.io/wdiv-scrapers/DC-PollingStations-Ealing

Thanks

83

Thanks for raising this @chris48s :star:

It seems like when I had to give morph a big hug yesterday the some of the scheduled runs didn’t quite make it through and got into a weird state where they were queued but not on the retries queue.

I found these Runs on the Rails console and set them running with:

Run.where(started_at: nil, finished_at: nil).where("queued_at < ?", 1.day.ago).where('scraper_id IS NOT NULL').each { |run| RunWorker.perform_async(run.id) }

@tmtmtmtm a bunch of yours and a bunch of PlanningAlerts scrapers where in there too. Those are running through now.

84

This is much less common these days, but I’ve got another one stuck: https://morph.io/wdiv-scrapers/DC-PollingStations-Portsmouth
I guess there may be one or 2 others from other users that also need a bit of a kick.
Thanks
Chris

85

Yikes! Thanks Chris. I’ve just found that 63 scrapers are in this states :frowning:

I did:

> Run.where(started_at: nil, finished_at: nil).where("queued_at < ?", 1.day.ago).where('scraper_id IS NOT NULL').map {|r| r.scraper.full_name }

On the rails console to get their names and indeed your’s is there.

There’s 62 scrapers in the “Unfinished runs attached to scrapers that do not have jobs on the queue” list when you run:

deploy@li421-88:/var/www/current$ cd /var/www/current && /home/deploy/.rvm/bin/rvm . do bundle exec rake app:emergency:show_queue_run_inconsistencies RAILS_ENV=production
Getting information from sidekiq queue...
Getting information about current containers...
The following runs do not have jobs on the queue:
[]
Unfinished runs attached to scrapers that do not have jobs on the queue:
[832949, 832950, 832951, 832952, 832953, 832955, 832956, 832957, 832958, 832959, 832960, 832961, 832962, 832963, 832964, 832965, 832966, 832967, 832968, 832969, 832970, 832971, 832972, 832973, 832974, 832975, 832976, 832977, 832978, 832979, 832980, 832981, 832982, 832983, 832984, 832985, 832986, 832987, 832988, 832989, 832990, 832991, 832992, 832993, 832994, 832995, 832996, 832997, 832998, 832999, 833000, 833001, 833002, 833003, 833004, 833005, 833006, 833007, 833008, 833009, 833010, 833011, 833012]

I’ll kick all these into action with:

[run id array from above].each{|id| RunWorker.perform_async(id)}

They’re now flowing through the queue states so I’ll keep an eye on it.

Thanks again, and good luck with election!

Luke

86

Thanks for sorting that out :slight_smile:

good luck with election!

Cheers. Polling day for the local elections was today. WhereDoIVote had rock solid uptime all day :tada: . One of my scrapers even helpfully notified me of a last minute polling station change this morning - so that was excellent.

…of course, whereas a few weeks ago we were expecting to be kicking back at this point, now the pressure is really on for the emergency general elections in a month :scream:

2 Likes
87

I think this happened because of the recent server issue and reboot :arrows_counterclockwise:

@equivalentideas something else for us to think about whenever we have to recover the machine and get things moving again :white_check_mark:

1 Like
88

Any chance of giving this one a kick: https://morph.io/wdiv-scrapers/DC-PollingStations-TonbridgeMalling
Bit of an unusual one this - it seems to have restarted itself several times before getting stuck.

Thanks
Chris

89

Hey!

mine is stuck too - https://morph.io/45h3r/rightmove

cheers

90

@45h3r, big upgrade tomorrow… and it should resolve couple big ticket items