Error with saving to sqlite - Ruby

1

I’ve got a weird problem with a scraper. It works except for the the save to sqlite line. Not quire sure why it is returning an error, and I couldn’t find the solution online. The syntax seems fine from other comparable scrapers.

Scraper:
https://morph.io/edmundtadros/top_250_IMDB_movies

Line:
ScraperWiki.save_sqlite([:rank], movie)

Error:
/app/vendor/bundle/ruby/2.0.0/gems/sqlite3-1.3.10/lib/sqlite3/statement.rb:39:in `bind_param’: can’t prepare Array (RuntimeError)

Repo: https://github.com/edmundtadros/top_250_IMDB_movies/blob/master/scraper.rb

2

Hey @edmundtadros as I understand it, SQLite will not accept values that are an Array. Here’s info about the datatypes that SQLite handles.

In your scraper.rb, on line 27, you’re using the split method, which returns an Array, to get a value for the movie_name attribute:

movie_name: tr.at('.titleColumn a').text.split(/"/)

See if trying to save a String instead helps.

1 Like
3

@edmundtadros did you get this sorted?

1 Like
4

Hello, yes I did sort it after a bit of trail and error.

I used the notation here to convert my array to a string.

So this: year: tr.at('.secondaryInfo').text.split(/[()]/)[1]
Became: year: [tr.at('.secondaryInfo').text.split(/[()]/)[1]]*","

5

Good to hear you got it sorted.

I don’t think that code is doing what you think it’s doing. You don’t need to convert an Array into a String, just select the element you want from it (in this case, the first and only element).

So @equivalentideas’s code:

movie_name: tr.at('.titleColumn a').text.split(/"/)

Becomes:

movie_name: tr.at('.titleColumn a').text.split(/"/).first

I’ve opened a pull request with some changes and some detailed commit messages of what I did and why. If you have any questions about it please fire away! :fire:

P.S. This is a great little scraper @edmundtadros. Adding more interesting data for people to use to morph.io - nice one!

6

Thanks Henare. Appreciate the help. I did get a bit confused with the array to string…

7

Any links to give me an idea of how to click through each page? I looked at some of the documentation for mechanise and it didn’t really enlighten much.

8

@edmundtadros here’s an example scraper I did for the workshops to show clicking through pages: