Get help with morph.io and scraping

Nothing saving to database

#41

Allo. It looks like nothing is saving again - my scraper ran a few hours ago with nothing saved :frowning:

#42

Scraping looks more reliable recently, but today I am having the nothing saving to database issue again. Sorry!

#43

Hey @lowndsy thanks for reporting this :slight_smile: Iā€™ve just run the tests scrapers and itā€™s not having trouble. Can you confirm this is still an issue?

#44

I had this one happen again today with https://morph.io/everypolitician-scrapers/azerbaijan-constituencies

#45

Iā€™m pretty sure itā€™s also happening with https://morph.io/everypolitician-scrapers/uk-house-of-commons-wikidata, as itā€™s been providing us data that is known to be stale, whilst reporting ā€œnothing changedā€

#46

Iā€™ve also noticed a few of mine doing this sporadically, but there doesnā€™t seem to be a pattern to it.

#47

Thanks for letting us know @tmtmtmtm and @chris48s . Iā€™ve restarted the service and confirmed that records are saving with the test scraper. https://github.com/openaustralia/morph/issues/1064

Let us know if this is still occurring :thumbsup:

#48

This has been happening again a lot over the last couple of days. https://morph.io/tmtmtmtm/south-african-national-assembly-members-wikidata is one current example.

#49

@tmtmtmtm this seems to be ok again. The test scraper is working.

If you see this happening, the current fix is to run the test scraper (you could fork it to run if you like), and if itā€™s not saving anything to the db, restart the morph service with the deploy user on the server.

sudo service morph restart
#50

@equivalentideas nope - still happening with some scrapers, e.g. https://morph.io/everypolitician-scrapers/uk-house-of-commons-wikidata. Lots of others are working though, so I donā€™t think itā€™s enough to check a single test scraper to see if itā€™s working. The problem is more subtle than thatā€¦

#51

@tmtmtmtm hmmmm. Thatā€™s looks to be running fine on itā€™s latest run. Any thoughts or more data you have would be great on the issue https://github.com/openaustralia/morph/issues/1064

#52

I think Iā€™ve seen this problem only affect one Sidekiq process (there are two in our current setup) so that might have been what was happening here.

#53

Same things again https://morph.io/achasik/Test2

#54

@achasik Iā€™ve just restarted the service so hopefully itā€™s fixed.

#55

This is happening a lot again today

#56

Yeah Iā€™ve been seeing it a fair bit over the last few days, but Iā€™d say most of my jobs have failed to write any data today

#57

Thanks for letting us know @tmtmtmtm and @chris48s Iā€™ve restarted the service and it should be working again. Let me know if it needs another kick.

#58

Cheers. All the ones Iā€™ve checked today have saved data :+1:

#59

This is failing fairly consistently again now.

#60

Looks like itā€™s working after yesterdayā€™s reset. Let me know if it needs attention :slight_smile: