Nothing saving to database


#66

I ran a test scraper which confirmed the problem. I then stopped the sidekiq processes from the admin backend and upstart subsequently started them back up. I ran a test scraper again and confirmed it was fixed :thumbsup:


#67

I didn’t realise you could do this in the admin. That’s brilliant. @tmtmtmtm you have access to that right? Wanna just jump in and give that a go next time and let us know how it goes? Still a good idea to report here and/or in Slack that you’re gonna do it first :slight_smile:


#68

Testing my scraper and nothing saved to database https://morph.io/LoveMyData/burwood_development_applications


#69

Thanks @LoveMyData I confirmed the bug on the test scraper and restarted the morph service. I’ve confirmed that it’s writing again with the test scraper. Could you check your one now?


#70

I’ve noticed quite a few scrapers failing to write data again over the last day. Any chance of a restart?


#71

Thanks @chris48s . I checked the test scraper and it seems to be back in action now. If it comes up again let us know and we’ll use @henare’s instructions from Slack to restart the worker:

I just realised that because you have access to the admin backend you can restart sidekiq when that happens Nothing saving to database Just click Stop all on this page https://morph.io/admin/jobs/busy?poll=true which will stop sidekiq and then upstart will recover it. I just tested that works :thumbsup::skin-tone-3:


#72

Unfortunately I still have several reporting that nothing changed in the database since that was posted. e.g:
https://morph.io/wdiv-scrapers/DC-PollingStations-Pembrokeshire
https://morph.io/wdiv-scrapers/DC-PollingStations-Southampton
https://morph.io/wdiv-scrapers/DC-PollingStations-Leeds
https://morph.io/wdiv-scrapers/DC-PollingStations-Wolverhampton
All of those should have written data as they should log to the history table on each run.


#73

@chris48s This is probably the situation where 1 of the 2 workers is affect, which is why @equivalentideas couldn’t see the problem with the test scraper. I’ve restarted the workers so hopefully it’s fixed now.


#74

Excellent - all the scrapers I’ve checked since you posted that report that they have written data. Thanks


Morph has fallen over (disk i/o does nothing)
#75

Happening again https://morph.io/LoveMyData/saplanningportal


9 records added, 9 records removed
#76

Thanks @LoveMyData.

I wasn’t seeing it on a test scraper but that probably means it ran on the Sidekiq process not affected.

I’ve restarted Sidekiq anyhow and that should have fixed it.


9 records added, 9 records removed
New data not added to database after clearing it
#77

Same problem here: https://morph.io/faloi/buscador-alquileres

Tested locally and works fine.


#78

Yeah - some of mine are failing too.


#79

I have been having the same issue! The computer just ignores the scraperwiki.sqlite.save function.

Also all scraperwiki functions seem to not be working. For example:

import scraperwiki
html = scraperwiki.scrape("http://www.AnyURLYouWantToScrapeGoesHere.com)
print html

Just scrapes the website of Columbia University’s computer science department, not the URL you put in there. Which is just bizarre. Please help!

-Lucia


#80

Thanks @walinchus @chris48s and @faloi . I just ran the test scraper to confirm that yep, it’s not saving.

I’ve restarted the sidekiq processes and checked that it’s working again :thumbsup:

It’s been a while, so here’s the link to the bug in Github in case anyone wants to have a go at it :slight_smile: https://github.com/openaustralia/morph/issues/1064

Thanks again for letting us know :slight_smile:

Luke


#81

I’m so sorry to bother you, I know you are busy. But if you could do that reboot again I would appreciate it. Things are not saving to the database again. I would try to fix the bug if I could, though I’m sure I would cause more problems. Thanks @equivalentideas! -Lucia


#82

Thanks so much! Working again!


#83

Mods - sorry to bother you again with his, but I’m getting the same error again across two scrapers. I noticed a few other scrapers are not saving anything to databases either. Is another reboot needed?


#84

Thanks for pinging @abgtrevize :+1: A queue restart was done ~15 hours ago by @tmtmtmtm so that should have fixed your problem too.


#85

Thanks @henare! It’s fixed now.