Nothing saving to database

@equivalentideas — this seems to have been fine yesterday, but happening again on anything running this morning :frowning:

:disappointed: I’ll restart the service.

Happening again now (spotted initially on

Thanks Tony. I’ve restarted the service.

This is back again today

I ran a test scraper which confirmed the problem. I then stopped the sidekiq processes from the admin backend and upstart subsequently started them back up. I ran a test scraper again and confirmed it was fixed :thumbsup:

1 Like

I didn’t realise you could do this in the admin. That’s brilliant. @tmtmtmtm you have access to that right? Wanna just jump in and give that a go next time and let us know how it goes? Still a good idea to report here and/or in Slack that you’re gonna do it first :slight_smile:

Testing my scraper and nothing saved to database

Thanks @LoveMyData I confirmed the bug on the test scraper and restarted the morph service. I’ve confirmed that it’s writing again with the test scraper. Could you check your one now?

I’ve noticed quite a few scrapers failing to write data again over the last day. Any chance of a restart?

Thanks @chris48s . I checked the test scraper and it seems to be back in action now. If it comes up again let us know and we’ll use @henare’s instructions from Slack to restart the worker:

I just realised that because you have access to the admin backend you can restart sidekiq when that happens Nothing saving to database - #66 by henare Just click Stop all on this page which will stop sidekiq and then upstart will recover it. I just tested that works :thumbsup::skin-tone-3:

Unfortunately I still have several reporting that nothing changed in the database since that was posted. e.g:
All of those should have written data as they should log to the history table on each run.

@chris48s This is probably the situation where 1 of the 2 workers is affect, which is why @equivalentideas couldn’t see the problem with the test scraper. I’ve restarted the workers so hopefully it’s fixed now.

Excellent - all the scrapers I’ve checked since you posted that report that they have written data. Thanks


Happening again

Thanks @LoveMyData.

I wasn’t seeing it on a test scraper but that probably means it ran on the Sidekiq process not affected.

I’ve restarted Sidekiq anyhow and that should have fixed it.

Same problem here:

Tested locally and works fine.

Yeah - some of mine are failing too.

I have been having the same issue! The computer just ignores the function.

Also all scraperwiki functions seem to not be working. For example:

import scraperwiki
html = scraperwiki.scrape("
print html

Just scrapes the website of Columbia University’s computer science department, not the URL you put in there. Which is just bizarre. Please help!


Thanks @walinchus @chris48s and @faloi . I just ran the test scraper to confirm that yep, it’s not saving.

I’ve restarted the sidekiq processes and checked that it’s working again :thumbsup:

It’s been a while, so here’s the link to the bug in Github in case anyone wants to have a go at it :slight_smile:

Thanks again for letting us know :slight_smile: