Nothing saving to database

Question & Answer
61

@equivalentideas — this seems to have been fine yesterday, but happening again on anything running this morning :frowning:

62

:disappointed: I’ll restart the service.

63

Happening again now (spotted initially on https://morph.io/tmtmtmtm/belarus-house-of-representatives-wikidata)

64

Thanks Tony. I’ve restarted the service.

65

This is back again today

66

I ran a test scraper which confirmed the problem. I then stopped the sidekiq processes from the admin backend and upstart subsequently started them back up. I ran a test scraper again and confirmed it was fixed :thumbsup:

1 Like
67

I didn’t realise you could do this in the admin. That’s brilliant. @tmtmtmtm you have access to that right? Wanna just jump in and give that a go next time and let us know how it goes? Still a good idea to report here and/or in Slack that you’re gonna do it first :slight_smile:

68

Testing my scraper and nothing saved to database https://morph.io/LoveMyData/burwood_development_applications

69

Thanks @LoveMyData I confirmed the bug on the test scraper and restarted the morph service. I’ve confirmed that it’s writing again with the test scraper. Could you check your one now?

70

I’ve noticed quite a few scrapers failing to write data again over the last day. Any chance of a restart?

71

Thanks @chris48s . I checked the test scraper and it seems to be back in action now. If it comes up again let us know and we’ll use @henare’s instructions from Slack to restart the worker:

I just realised that because you have access to the admin backend you can restart sidekiq when that happens Nothing saving to database - #66 by henare Just click Stop all on this page morph.io which will stop sidekiq and then upstart will recover it. I just tested that works :thumbsup::skin-tone-3:

72

Unfortunately I still have several reporting that nothing changed in the database since that was posted. e.g:
https://morph.io/wdiv-scrapers/DC-PollingStations-Pembrokeshire
https://morph.io/wdiv-scrapers/DC-PollingStations-Southampton
https://morph.io/wdiv-scrapers/DC-PollingStations-Leeds
https://morph.io/wdiv-scrapers/DC-PollingStations-Wolverhampton
All of those should have written data as they should log to the history table on each run.

73

@chris48s This is probably the situation where 1 of the 2 workers is affect, which is why @equivalentideas couldn’t see the problem with the test scraper. I’ve restarted the workers so hopefully it’s fixed now.

74

Excellent - all the scrapers I’ve checked since you posted that report that they have written data. Thanks

2 Likes
75

Happening again https://morph.io/LoveMyData/saplanningportal

76

Thanks @LoveMyData.

I wasn’t seeing it on a test scraper but that probably means it ran on the Sidekiq process not affected.

I’ve restarted Sidekiq anyhow and that should have fixed it.

77

Same problem here: https://morph.io/faloi/buscador-alquileres

Tested locally and works fine.

78

Yeah - some of mine are failing too.

79

I have been having the same issue! The computer just ignores the scraperwiki.sqlite.save function.

Also all scraperwiki functions seem to not be working. For example:

import scraperwiki
html = scraperwiki.scrape("http://www.AnyURLYouWantToScrapeGoesHere.com)
print html

Just scrapes the website of Columbia University’s computer science department, not the URL you put in there. Which is just bizarre. Please help!

-Lucia

80

Thanks @walinchus @chris48s and @faloi . I just ran the test scraper to confirm that yep, it’s not saving.

I’ve restarted the sidekiq processes and checked that it’s working again :thumbsup:

It’s been a while, so here’s the link to the bug in Github in case anyone wants to have a go at it :slight_smile: https://github.com/openaustralia/morph/issues/1064

Thanks again for letting us know :slight_smile:

Luke

2 Likes