My scraper stuck, what to do?

Question & Answer
59

Another unstoppable beast here: https://morph.io/PuZZleDucK/VIC_Tenders

I’m sure this one might be my scrapers fault as there was (still is) lots of crazy session resetting… just realized I wasn’t quitting all of the sessions, pushing an update to fix that now

A gentle kick would be much appreciated

60

Looks unstuck to me now :slight_smile: We’ve been having some issues with the queue the last few days, but it seems to be working pretty well at the moment :thumbsup:

61

Hi. Haven’t raised it over the holidays, but I’ve got a lot of stuck scrapers under the wdiv-scrapers org: https://morph.io/wdiv-scrapers
Some of them have been queued for several weeks now. Any chance of getting them cleared?
Thanks

62

bumping as it has been a week - sorry to hassle

63

@chris48s nice hassling like this is no hassle :slight_smile:

I’ve been away on holidays but @equivalentideas has done a fantastic job of responding to queue and disk space issues with morph.io while I’ve been away - thank you Luke :green_heart:

After another round of issues over the last couple of days we both sat down and purged a bunch of Docker cruft that should have resolved all these issues.

We’ve still got a stack of problems that will recur, including the most mundane - disk space, but we’ll try to keep on top of it.

64

I’ve got a load of scrapers that have been stuck in the queue for 5 days or so:
https://morph.io/wdiv-scrapers/DC-PollingStations-Mid-Sussex
https://morph.io/wdiv-scrapers/DC-PollingStations-Guildford
https://morph.io/wdiv-scrapers/DC-PollingStations-Kingston
https://morph.io/wdiv-scrapers/DC-PollingStations-Erewash
https://morph.io/wdiv-scrapers/DC-PollingStations-Daventry
https://morph.io/wdiv-scrapers/DC-PollingStations-Wolverhampton
https://morph.io/wdiv-scrapers/DC-PollingStations-Hackney
I suspect when you cut them free, they’ll probably start failing with 128, but it would be useful to get them trying again at least (and free up the containers).

65

So many problems. So little time.

I’ve called stop on all of these. They should say failed with exit code 255 and now be runnable, including by autorun.

I’m not around for a few days but we have a new friend looking at the problems with us now, and Luke will still be keeping a watchful eye on things.

66

Things seem to have been more stable of late, but I’ve got a few that have been dead for a week or so now:
https://morph.io/wdiv-scrapers/DC-PollingStations-Lambeth
https://morph.io/wdiv-scrapers/DC-PollingStations-Southampton
https://morph.io/wdiv-scrapers/DC-PollingStations-Barnet

Any chance of giving them a kick?
Thanks

67

I’ve stopped those three on the Rails console with:

Scraper.find_by(full_name: "wdiv-scrapers/DC-PollingStations-Barnet").stop!

Let us know if this pops up again @chris48s :slight_smile:

68

Looking at this I found a bunch of other scrapers in this state and stopped them:

woodbine/sp_E1702_SCC_gov
duncanparkes/papuanewguinea
yotam-gafni/openbudget_israel
woodbine/sp_DFT004_DFT_gov
planningalerts-scrapers/liverpool_plains
tmtmtmtm/afghanistan-assembly
woodbine/CQC
everypolitician-scrapers/russia-duma-2016
colsondonohue/pitchfork_review_data
tmtmtmtm/romanian-parliament
soit-sk/scraper-sk_company_register
CodeCompiler/facebook_profile_scraper
tmtmtmtm/kenya-mzalendo
danishthanvi1/my-cz_cmp_list
alisonkeen/sa_lg_councillors

69

Thanks for killing those ones @equivalentideas, but it looks like there are about 40 others in tmtmtmtm and everypolitician-scrapers stuck for almost a week (e.g. tmtmtmtm/moldova_parlament, tmtmtmtm/montenegro-parldata, everypolitician-scrapers/samoa-parliament, https://morph.io/everypolitician-scrapers/croatian-parliament-wikidata). Any clue why those aren’t coming up in your search, and how to detect and autofix things like this?

70

Thanks for letting me know @tmtmtmtm ! I found 116 scrapers in this state, including a bunch of yours :crying_cat_face:

I tracked down the problem and set them all off to run. You can see a bit of the story here: Scrapers can appear to run for more than 24 hours · Issue #1050 · openaustralia/morph · GitHub

I’ll keep an eye on them. We should see them all run through over the next few hours.

Any clue why those aren’t coming up in your search, and how to detect and autofix things like this?

I traced back the problem here Scrapers can appear to run for more than 24 hours · Issue #1050 · openaustralia/morph · GitHub and I’m pretty sure the root cause is Set noeviction policy on redis · Issue #1056 · openaustralia/morph · GitHub .

When we were having all the disk space issues last week, Redis probably dropped a whole bunch of jobs, which left these Runs stuck in a queued but not running state, which avoids all our different lists.

I think fixing that will fix this issue, but we’re also out of the woods with the disk space nightmare, so hopefully wont see this for a while anyway: Docker is exceeding it's disk space allocation, gradually exhausting all free space · Issue #1104 · openaustralia/morph · GitHub

1 Like
71

Thanks @equivalentideas

Looks like https://morph.io/everypolitician-scrapers/croatia-parliament is also stuck and getting get fixed up with this. It gave an interesting warning/error that might help point at something:

Error removing intermediate container 4021d85103a9: Driver aufs failed to remove root filesystem 4021d85103a9a225394329d397cb1c07619b0ff333be923c179f413f4b01d708: rename /var/lib/docker/aufs/mnt/a31e157bcb10e77bb626c757d25fbdf0be9972da6358db738030a08dd0393d97 /var/lib/docker/aufs/mnt/a31e157bcb10e77bb626c757d25fbdf0be9972da6358db738030a08dd0393d97-removing: device or resource busy
Injecting scraper and running…

72

That rerunning behaviour looks like this to me https://github.com/openaustralia/morph/issues/1087

I’ll call stop! on it.

73

That successfully stopped it :+1: I got that solution here https://github.com/openaustralia/morph/issues/1087#issuecomment-247217293

74

It’s similar behaviour, but a different cause we think. @henare has just pointed out that the docker prune we’re running is removing containers attached to runs in the retry queue that have stopped (see Scrapers with no console output can take an unnecessarily long time to finish running · Issue #1123 · openaustralia/morph · GitHub for why the containers stop).

The prune is removing the container, which may be causing the scraper to start from scratch. Which is why it appears to be re-running again and again when it hits the front of the queue.

75

The prune is removing the container, which may be causing the scraper to start from scratch. Which is why it appears to be re-running again and again when it hits the front of the queue.

There’s now an issue for this Docker pruning is busting image cache and possibly destroying containers in use · Issue #1124 · openaustralia/morph · GitHub

76

This is happening way less frequently at the moment, but I’ve got another one which has been stuck for a few days: https://morph.io/wdiv-scrapers/DC-PollingStations-Halton

77

May have spoken a bit too soon there :frowning: Can you just kill off anything under https://morph.io/wdiv-scrapers that has died?
Cheers

78

Got a few reporting that’ve been queued for over a day:
https://morph.io/wdiv-scrapers/DC-PollingStations-Calderdale
https://morph.io/wdiv-scrapers/DC-PollingStations-Dover
https://morph.io/wdiv-scrapers/DC-PollingStations-CityOfLondon

When someone gets to having a look at it, if there’s anything else in the https://morph.io/wdiv-scrapers org that has become stuck by that point (I’ve got a few others that have been in the queue for 10+ hours at this point but may still run or fail, I guess), can you give them all a spring clean.
Thanks