Nothing saving to database

Question & Answer
#32

I’m having the same problem with this scraper: https://morph.io/everypolitician-scrapers/guernsey-2016

#33

@ondenman hmm, not seeing the problem with the test scraper this time but I’ve restarted the queue anyhow.

#34

Sorry, looks like it is back - nothing saved to database for my last two scrapes.

#35

Oops, nothing saved too to database for ukraine_verkhovna_rada_votes more details https://github.com/openaustralia/ukraine_verkhovna_rada_votes/issues/17

#36

nothing changed in the database again

#37

We’ve fixed that problem and also addressed the big backlog of scrapers that started happening about 3 days ago.

You can read what we did on the issue: https://github.com/openaustralia/morph/issues/1064#issuecomment-255626498

There is still a backlog of over 600 runs right now but morph.io is working its way through them.

My scraper stuck, what to do?
#38

The problem still remain

#39

You’re right @achasik, thanks for letting us know. I’ve updated the issue with what I’ve been doing to fix it https://github.com/openaustralia/morph/issues/1064#issuecomment-255895896

#40

Good Morning Gents. It looks like things aren’t saving again - failed over the weekend and I have just re-ran with the same result.

#41

Allo. It looks like nothing is saving again - my scraper ran a few hours ago with nothing saved :frowning:

#42

Scraping looks more reliable recently, but today I am having the nothing saving to database issue again. Sorry!

#43

Hey @lowndsy thanks for reporting this :slight_smile: I’ve just run the tests scrapers and it’s not having trouble. Can you confirm this is still an issue?

#44

I had this one happen again today with https://morph.io/everypolitician-scrapers/azerbaijan-constituencies

#45

I’m pretty sure it’s also happening with https://morph.io/everypolitician-scrapers/uk-house-of-commons-wikidata, as it’s been providing us data that is known to be stale, whilst reporting “nothing changed”

#46

I’ve also noticed a few of mine doing this sporadically, but there doesn’t seem to be a pattern to it.

#47

Thanks for letting us know @tmtmtmtm and @chris48s . I’ve restarted the service and confirmed that records are saving with the test scraper. https://github.com/openaustralia/morph/issues/1064

Let us know if this is still occurring :thumbsup:

#48

This has been happening again a lot over the last couple of days. https://morph.io/tmtmtmtm/south-african-national-assembly-members-wikidata is one current example.

#49

@tmtmtmtm this seems to be ok again. The test scraper is working.

If you see this happening, the current fix is to run the test scraper (you could fork it to run if you like), and if it’s not saving anything to the db, restart the morph service with the deploy user on the server.

sudo service morph restart
#50

@equivalentideas nope - still happening with some scrapers, e.g. https://morph.io/everypolitician-scrapers/uk-house-of-commons-wikidata. Lots of others are working though, so I don’t think it’s enough to check a single test scraper to see if it’s working. The problem is more subtle than that…

#51

@tmtmtmtm hmmmm. That’s looks to be running fine on it’s latest run. Any thoughts or more data you have would be great on the issue https://github.com/openaustralia/morph/issues/1064