Hi there,
This scraper of mine seems to be inexplicably failing with status 255. I’ve seen lots of other threads on this previously, but they seem to point to this issue, which doesn’t look relevant.
Any help would be appreciated, cheers!
Hi @mashedkeyboard!
Status 255 indicates that the container terminated unexpectedly. In cases like the one you referenced that has happened because of systemic issues with Morph, which is why it was occurring to lots of scrapers then.
In this case, I’ve looked at your scraper and been able to figure out that it’s failing because it’s hitting the 512Mb memory limit and being terminated. I can see from this commit:
that this is a problem you’ve run into before. I’ve done a quick test and it looks like decreasing the batch size further does help the memory usage - I was able to complete a run successfully with a batch size of 25 (see https://morph.io/jamezpolley/uk-house-of-commons-wikidata) which was sitting around the 425Mb mark for most of the run.