It scrapes the data fine, which I can see when I print data, but nothing is being saved to the SQL database when I use the scraperwiki function. I have compared this to another scraper I have that works and I can’t see how they are different and why this one isn’t working! Can anyone help?
Apologies. Didn’t notice the ping…
Think I found why it wasn’t saving your data, @Abbie2020. Here’s one formatted sample record from the scraper:
{
'source': 'http://www.reading.ac.uk/15/about/newsandevents/Events/Event659657.aspx',
'contact': [
'School Direct (Primary and Secondary) sxxxxxx@xxxx.ac.uk\r',
'PGCE Primary pxxxxxx@xxxx.ac.uk\r',
'PGCE Secondary pxxxxxx@xxxx.ac.uk \r'
],
'description': [
'The Institute of Education is delighted to present ....',
'\r',
'Admission free, all welcome. Booking required.\r',
'\r',
'\r'
],
'title': 'Institute of Education routes into teaching information evenings',
'date': '2016-04-11',
'time': '16:00',
'venue': 'Institute of Education',
'id': 'uor659657',
'location': 'Institute of Education, London Road campus'
}
You’ll notice the contact and description fields have list values. Scraperwiki’s sql.save() function expects each field to have a simple, atomic value (eg. int, string, float…).
I’m not sure why the save routine just failed silently though… You’d imagine it should explode or something, rather than just skip saving the record. Can’t really explain that part, sorry.
I’ve shot up one way of getting the data saving, over here. Will shoot a pull request across, but feel free to ignore if you’ve got different requirements on handling the data.
1 Like
Hi @otherchirps, thank you very much for helping me! I’m just learning as I go with this so I really appreciate you spending the time looking into it.
Your solution worked. It’s not obvious so I would never have spotted it! Now I know how to tweak it if I want to change anything.
Thanks again 
1 Like