Get help with morph.io and scraping

Nothing saving to database

lowndsy 2016-12-21 14:16:57 UTC #41

Allo. It looks like nothing is saving again - my scraper ran a few hours ago with nothing saved

lowndsy 2017-01-31 14:49:27 UTC #42

Scraping looks more reliable recently, but today I am having the nothing saving to database issue again. Sorry!

equivalentideas 2017-02-01 09:00:39 UTC #43

Hey @lowndsy thanks for reporting this I've just run the tests scrapers and it's not having trouble. Can you confirm this is still an issue?

tmtmtmtm 2017-02-05 09:24:29 UTC #44

I had this one happen again today with https://morph.io/everypolitician-scrapers/azerbaijan-constituencies

tmtmtmtm 2017-02-05 11:04:49 UTC #45

I'm pretty sure it's also happening with https://morph.io/everypolitician-scrapers/uk-house-of-commons-wikidata, as it's been providing us data that is known to be stale, whilst reporting "nothing changed"

chris48s 2017-02-05 21:12:27 UTC #46

I've also noticed a few of mine doing this sporadically, but there doesn't seem to be a pattern to it.

equivalentideas 2017-02-06 04:09:07 UTC #47

Thanks for letting us know @tmtmtmtm and @chris48s . I've restarted the service and confirmed that records are saving with the test scraper. https://github.com/openaustralia/morph/issues/1064

Let us know if this is still occurring

tmtmtmtm 2017-02-23 13:19:40 UTC #48

This has been happening again a lot over the last couple of days. https://morph.io/tmtmtmtm/south-african-national-assembly-members-wikidata is one current example.

equivalentideas 2017-02-23 23:43:58 UTC #49

@tmtmtmtm this seems to be ok again. The test scraper is working.

If you see this happening, the current fix is to run the test scraper (you could fork it to run if you like), and if it's not saving anything to the db, restart the morph service with the deploy user on the server.

sudo service morph restart

tmtmtmtm 2017-02-24 10:27:52 UTC #50

@equivalentideas nope - still happening with some scrapers, e.g. https://morph.io/everypolitician-scrapers/uk-house-of-commons-wikidata. Lots of others are working though, so I don't think it's enough to check a single test scraper to see if it's working. The problem is more subtle than that…

equivalentideas 2017-02-27 01:24:23 UTC #51

@tmtmtmtm hmmmm. That’s looks to be running fine on it's latest run. Any thoughts or more data you have would be great on the issue https://github.com/openaustralia/morph/issues/1064

henare 2017-02-27 02:27:55 UTC #52

I think I've seen this problem only affect one Sidekiq process (there are two in our current setup) so that might have been what was happening here.

achasik 2017-03-03 06:25:07 UTC #53

Same things again https://morph.io/achasik/Test2

henare 2017-03-06 02:49:54 UTC #54

@achasik I've just restarted the service so hopefully it's fixed.

tmtmtmtm 2017-03-30 15:06:19 UTC #55

This is happening a lot again today

chris48s 2017-03-30 20:32:27 UTC #56

Yeah I've been seeing it a fair bit over the last few days, but I'd say most of my jobs have failed to write any data today

equivalentideas 2017-03-30 22:11:39 UTC #57

Thanks for letting us know @tmtmtmtm and @chris48s I've restarted the service and it should be working again. Let me know if it needs another kick.

chris48s 2017-03-31 15:04:00 UTC #58

Cheers. All the ones I've checked today have saved data

tmtmtmtm 2017-04-02 20:43:20 UTC #59

This is failing fairly consistently again now.

equivalentideas 2017-04-03 01:24:56 UTC #60

Looks like it's working after yesterday's reset. Let me know if it needs attention