Testing my scraper and nothing saved to database https://morph.io/LoveMyData/burwood_development_applications
Thanks @LoveMyData I confirmed the bug on the test scraper and restarted the morph service. I’ve confirmed that it’s writing again with the test scraper. Could you check your one now?
I’ve noticed quite a few scrapers failing to write data again over the last day. Any chance of a restart?
I just realised that because you have access to the admin backend you can restart sidekiq when that happens Nothing saving to database Just click Stop all on this page https://morph.io/admin/jobs/busy?poll=true which will stop sidekiq and then upstart will recover it. I just tested that works :skin-tone-3:
Unfortunately I still have several reporting that nothing changed in the database since that was posted. e.g:
All of those should have written data as they should log to the
history table on each run.
@chris48s This is probably the situation where 1 of the 2 workers is affect, which is why @equivalentideas couldn’t see the problem with the test scraper. I’ve restarted the workers so hopefully it’s fixed now.
Excellent - all the scrapers I’ve checked since you posted that report that they have written data. Thanks
Happening again https://morph.io/LoveMyData/saplanningportal
I wasn’t seeing it on a test scraper but that probably means it ran on the Sidekiq process not affected.
I’ve restarted Sidekiq anyhow and that should have fixed it.
Same problem here: https://morph.io/faloi/buscador-alquileres
Tested locally and works fine.
Yeah - some of mine are failing too.
I have been having the same issue! The computer just ignores the scraperwiki.sqlite.save function.
Also all scraperwiki functions seem to not be working. For example:
html = scraperwiki.scrape("http://www.AnyURLYouWantToScrapeGoesHere.com)
Just scrapes the website of Columbia University’s computer science department, not the URL you put in there. Which is just bizarre. Please help!
I’ve restarted the sidekiq processes and checked that it’s working again
It’s been a while, so here’s the link to the bug in Github in case anyone wants to have a go at it https://github.com/openaustralia/morph/issues/1064
Thanks again for letting us know
I’m so sorry to bother you, I know you are busy. But if you could do that reboot again I would appreciate it. Things are not saving to the database again. I would try to fix the bug if I could, though I’m sure I would cause more problems. Thanks @equivalentideas! -Lucia
Thanks so much! Working again!
Mods - sorry to bother you again with his, but I’m getting the same error again across two scrapers. I noticed a few other scrapers are not saving anything to databases either. Is another reboot needed?
Thanks @henare! It’s fixed now.