My scraper stuck, what to do?


#62

bumping as it has been a week - sorry to hassle


#63

@chris48s nice hassling like this is no hassle :slight_smile:

I’ve been away on holidays but @equivalentideas has done a fantastic job of responding to queue and disk space issues with morph.io while I’ve been away - thank you Luke :green_heart:

After another round of issues over the last couple of days we both sat down and purged a bunch of Docker cruft that should have resolved all these issues.

We’ve still got a stack of problems that will recur, including the most mundane - disk space, but we’ll try to keep on top of it.


#64

I’ve got a load of scrapers that have been stuck in the queue for 5 days or so:
https://morph.io/wdiv-scrapers/DC-PollingStations-Mid-Sussex
https://morph.io/wdiv-scrapers/DC-PollingStations-Guildford
https://morph.io/wdiv-scrapers/DC-PollingStations-Kingston
https://morph.io/wdiv-scrapers/DC-PollingStations-Erewash
https://morph.io/wdiv-scrapers/DC-PollingStations-Daventry
https://morph.io/wdiv-scrapers/DC-PollingStations-Wolverhampton
https://morph.io/wdiv-scrapers/DC-PollingStations-Hackney
I suspect when you cut them free, they’ll probably start failing with 128, but it would be useful to get them trying again at least (and free up the containers).


Scraper using Splinter works locally but not on Morph
#65

So many problems. So little time.

I’ve called stop on all of these. They should say failed with exit code 255 and now be runnable, including by autorun.

I’m not around for a few days but we have a new friend looking at the problems with us now, and Luke will still be keeping a watchful eye on things.


Scraper using Splinter works locally but not on Morph
#66

Things seem to have been more stable of late, but I’ve got a few that have been dead for a week or so now:
https://morph.io/wdiv-scrapers/DC-PollingStations-Lambeth
https://morph.io/wdiv-scrapers/DC-PollingStations-Southampton
https://morph.io/wdiv-scrapers/DC-PollingStations-Barnet

Any chance of giving them a kick?
Thanks


#67

I’ve stopped those three on the Rails console with:

Scraper.find_by(full_name: "wdiv-scrapers/DC-PollingStations-Barnet").stop!

Let us know if this pops up again @chris48s :slight_smile:


#68

Looking at this I found a bunch of other scrapers in this state and stopped them:

woodbine/sp_E1702_SCC_gov
duncanparkes/papuanewguinea
yotam-gafni/openbudget_israel
woodbine/sp_DFT004_DFT_gov
planningalerts-scrapers/liverpool_plains
tmtmtmtm/afghanistan-assembly
woodbine/CQC
everypolitician-scrapers/russia-duma-2016
colsondonohue/pitchfork_review_data
tmtmtmtm/romanian-parliament
soit-sk/scraper-sk_company_register
CodeCompiler/facebook_profile_scraper
tmtmtmtm/kenya-mzalendo
danishthanvi1/my-cz_cmp_list
alisonkeen/sa_lg_councillors


#69

Thanks for killing those ones @equivalentideas, but it looks like there are about 40 others in tmtmtmtm and everypolitician-scrapers stuck for almost a week (e.g. tmtmtmtm/moldova_parlament, tmtmtmtm/montenegro-parldata, everypolitician-scrapers/samoa-parliament, https://morph.io/everypolitician-scrapers/croatian-parliament-wikidata). Any clue why those aren’t coming up in your search, and how to detect and autofix things like this?


#70

Thanks for letting me know @tmtmtmtm ! I found 116 scrapers in this state, including a bunch of yours :crying_cat_face:

I tracked down the problem and set them all off to run. You can see a bit of the story here: https://github.com/openaustralia/morph/issues/1050#issuecomment-277882446

I’ll keep an eye on them. We should see them all run through over the next few hours.

Any clue why those aren’t coming up in your search, and how to detect and autofix things like this?

I traced back the problem here https://github.com/openaustralia/morph/issues/1050#issuecomment-277894267 and I’m pretty sure the root cause is https://github.com/openaustralia/morph/issues/1056 .

When we were having all the disk space issues last week, Redis probably dropped a whole bunch of jobs, which left these Runs stuck in a queued but not running state, which avoids all our different lists.

I think fixing that will fix this issue, but we’re also out of the woods with the disk space nightmare, so hopefully wont see this for a while anyway: https://github.com/openaustralia/morph/issues/1104#issuecomment-276840504


#71

Thanks @equivalentideas

Looks like https://morph.io/everypolitician-scrapers/croatia-parliament is also stuck and getting get fixed up with this. It gave an interesting warning/error that might help point at something:

Error removing intermediate container 4021d85103a9: Driver aufs failed to remove root filesystem 4021d85103a9a225394329d397cb1c07619b0ff333be923c179f413f4b01d708: rename /var/lib/docker/aufs/mnt/a31e157bcb10e77bb626c757d25fbdf0be9972da6358db738030a08dd0393d97 /var/lib/docker/aufs/mnt/a31e157bcb10e77bb626c757d25fbdf0be9972da6358db738030a08dd0393d97-removing: device or resource busy
Injecting scraper and running…


#72

That rerunning behaviour looks like this to me https://github.com/openaustralia/morph/issues/1087

I’ll call stop! on it.


#73

That successfully stopped it :+1: I got that solution here https://github.com/openaustralia/morph/issues/1087#issuecomment-247217293


#74

It’s similar behaviour, but a different cause we think. @henare has just pointed out that the docker prune we’re running is removing containers attached to runs in the retry queue that have stopped (see https://github.com/openaustralia/morph/issues/1123 for why the containers stop).

The prune is removing the container, which may be causing the scraper to start from scratch. Which is why it appears to be re-running again and again when it hits the front of the queue.


#75

The prune is removing the container, which may be causing the scraper to start from scratch. Which is why it appears to be re-running again and again when it hits the front of the queue.

There’s now an issue for this https://github.com/openaustralia/morph/issues/1124


#76

This is happening way less frequently at the moment, but I’ve got another one which has been stuck for a few days: https://morph.io/wdiv-scrapers/DC-PollingStations-Halton


#77

May have spoken a bit too soon there :frowning: Can you just kill off anything under https://morph.io/wdiv-scrapers that has died?
Cheers


#78

Got a few reporting that’ve been queued for over a day:
https://morph.io/wdiv-scrapers/DC-PollingStations-Calderdale
https://morph.io/wdiv-scrapers/DC-PollingStations-Dover
https://morph.io/wdiv-scrapers/DC-PollingStations-CityOfLondon

When someone gets to having a look at it, if there’s anything else in the https://morph.io/wdiv-scrapers org that has become stuck by that point (I’ve got a few others that have been in the queue for 10+ hours at this point but may still run or fail, I guess), can you give them all a spring clean.
Thanks


#79

@chris48s we’ve been doing some server gardening and found a bit of a queue backlog. That’s clearing now so hopefully it’ll be :sunflower: very soon.


#80

server gardening

:computer: :seedling: :deciduous_tree:


#81

Thought I had one stuck, but it eventually got cleaned up :smiley: False alarm :alarm_clock: