Scraper won't stop running

jennahowe · September 16, 2015, 4:52pm

The ‘stop scraper’ button isn’t doing anything.

It’s been running for about a day now https://morph.io/jennahowe/El_Salvador_Legislative_Assembly, so I guess it will be killed automatically soon when it goes over 24 hours.

It should have just stopped of it’s own accord ages ago because it errored pretty much straight away (bad gateway error), but that hasn’t happened.

henare · September 16, 2015, 11:25pm

Thanks for reporting this @jennahowe

I’ve logged a bug and also manually fixed your scraper up.

jennahowe · September 17, 2015, 8:36am

Thanks @henare

katkad · October 28, 2015, 2:21am

please stop this one too https://morph.io/soit-sk/trademarks_-_upv_sk

henare · October 28, 2015, 2:23am

@katkad I’ve kicked off the stuck run and it’s trying to finish now.

katkad · October 28, 2015, 9:43am

hello, it is still stuck I cannot stop it.

henare · November 3, 2015, 9:27am

The last run of that scraper logged over 180,000 lines and scraped more than 120,000 pages.

I’d suggest not outputting so many lines as this makes your scraper page very big. It would also be a good idea to add some throttling to your scraper if you’re scraping tens of thousands of pages.