Nothing saving to database


I ran a test scraper which confirmed the problem. I then stopped the sidekiq processes from the admin backend and upstart subsequently started them back up. I ran a test scraper again and confirmed it was fixed :thumbsup:


I didn’t realise you could do this in the admin. That’s brilliant. @tmtmtmtm you have access to that right? Wanna just jump in and give that a go next time and let us know how it goes? Still a good idea to report here and/or in Slack that you’re gonna do it first :slight_smile:


Testing my scraper and nothing saved to database


Thanks @LoveMyData I confirmed the bug on the test scraper and restarted the morph service. I’ve confirmed that it’s writing again with the test scraper. Could you check your one now?


I’ve noticed quite a few scrapers failing to write data again over the last day. Any chance of a restart?


Thanks @chris48s . I checked the test scraper and it seems to be back in action now. If it comes up again let us know and we’ll use @henare’s instructions from Slack to restart the worker:

I just realised that because you have access to the admin backend you can restart sidekiq when that happens Nothing saving to database Just click Stop all on this page which will stop sidekiq and then upstart will recover it. I just tested that works :thumbsup::skin-tone-3:


Unfortunately I still have several reporting that nothing changed in the database since that was posted. e.g:
All of those should have written data as they should log to the history table on each run.


@chris48s This is probably the situation where 1 of the 2 workers is affect, which is why @equivalentideas couldn’t see the problem with the test scraper. I’ve restarted the workers so hopefully it’s fixed now.


Excellent - all the scrapers I’ve checked since you posted that report that they have written data. Thanks

Morph has fallen over (disk i/o does nothing)

Happening again

9 records added, 9 records removed

Thanks @LoveMyData.

I wasn’t seeing it on a test scraper but that probably means it ran on the Sidekiq process not affected.

I’ve restarted Sidekiq anyhow and that should have fixed it.

9 records added, 9 records removed
New data not added to database after clearing it

Same problem here:

Tested locally and works fine.


Yeah - some of mine are failing too.


I have been having the same issue! The computer just ignores the function.

Also all scraperwiki functions seem to not be working. For example:

import scraperwiki
html = scraperwiki.scrape("
print html

Just scrapes the website of Columbia University’s computer science department, not the URL you put in there. Which is just bizarre. Please help!



Thanks @walinchus @chris48s and @faloi . I just ran the test scraper to confirm that yep, it’s not saving.

I’ve restarted the sidekiq processes and checked that it’s working again :thumbsup:

It’s been a while, so here’s the link to the bug in Github in case anyone wants to have a go at it :slight_smile:

Thanks again for letting us know :slight_smile:



I’m so sorry to bother you, I know you are busy. But if you could do that reboot again I would appreciate it. Things are not saving to the database again. I would try to fix the bug if I could, though I’m sure I would cause more problems. Thanks @equivalentideas! -Lucia


Thanks so much! Working again!


Mods - sorry to bother you again with his, but I’m getting the same error again across two scrapers. I noticed a few other scrapers are not saving anything to databases either. Is another reboot needed?


Thanks for pinging @abgtrevize :+1: A queue restart was done ~15 hours ago by @tmtmtmtm so that should have fixed your problem too.


Thanks @henare! It’s fixed now.