Nothing saving to database

Testing my scraper and nothing saved to database https://morph.io/LoveMyData/burwood_development_applications

Thanks @LoveMyData I confirmed the bug on the test scraper and restarted the morph service. I’ve confirmed that it’s writing again with the test scraper. Could you check your one now?

I’ve noticed quite a few scrapers failing to write data again over the last day. Any chance of a restart?

Thanks @chris48s . I checked the test scraper and it seems to be back in action now. If it comes up again let us know and we’ll use @henare’s instructions from Slack to restart the worker:

I just realised that because you have access to the admin backend you can restart sidekiq when that happens Nothing saving to database Just click Stop all on this page https://morph.io/admin/jobs/busy?poll=true which will stop sidekiq and then upstart will recover it. I just tested that works :thumbsup::skin-tone-3:

Unfortunately I still have several reporting that nothing changed in the database since that was posted. e.g:
https://morph.io/wdiv-scrapers/DC-PollingStations-Pembrokeshire
https://morph.io/wdiv-scrapers/DC-PollingStations-Southampton
https://morph.io/wdiv-scrapers/DC-PollingStations-Leeds
https://morph.io/wdiv-scrapers/DC-PollingStations-Wolverhampton
All of those should have written data as they should log to the history table on each run.

@chris48s This is probably the situation where 1 of the 2 workers is affect, which is why @equivalentideas couldn’t see the problem with the test scraper. I’ve restarted the workers so hopefully it’s fixed now.

Excellent - all the scrapers I’ve checked since you posted that report that they have written data. Thanks

2 Likes

Happening again https://morph.io/LoveMyData/saplanningportal

Thanks @LoveMyData.

I wasn’t seeing it on a test scraper but that probably means it ran on the Sidekiq process not affected.

I’ve restarted Sidekiq anyhow and that should have fixed it.

Same problem here: https://morph.io/faloi/buscador-alquileres

Tested locally and works fine.

Yeah - some of mine are failing too.

I have been having the same issue! The computer just ignores the scraperwiki.sqlite.save function.

Also all scraperwiki functions seem to not be working. For example:

import scraperwiki
html = scraperwiki.scrape("http://www.AnyURLYouWantToScrapeGoesHere.com)
print html

Just scrapes the website of Columbia University’s computer science department, not the URL you put in there. Which is just bizarre. Please help!

-Lucia

Thanks @walinchus @chris48s and @faloi . I just ran the test scraper to confirm that yep, it’s not saving.

I’ve restarted the sidekiq processes and checked that it’s working again :thumbsup:

It’s been a while, so here’s the link to the bug in Github in case anyone wants to have a go at it :slight_smile: https://github.com/openaustralia/morph/issues/1064

Thanks again for letting us know :slight_smile:

Luke

2 Likes

I’m so sorry to bother you, I know you are busy. But if you could do that reboot again I would appreciate it. Things are not saving to the database again. I would try to fix the bug if I could, though I’m sure I would cause more problems. Thanks @equivalentideas! -Lucia

1 Like

Thanks so much! Working again!

1 Like

Mods - sorry to bother you again with his, but I’m getting the same error again across two scrapers. I noticed a few other scrapers are not saving anything to databases either. Is another reboot needed?

1 Like

Thanks for pinging @abgtrevize :+1: A queue restart was done ~15 hours ago by @tmtmtmtm so that should have fixed your problem too.

1 Like

Thanks @henare! It’s fixed now.

1 Like

Getting same issue @mlandauer with https://morph.io/codecakes/beds

See https://github.com/openaustralia/morph/issues/1064#issuecomment-664052034