New data not added to database after clearing it

Question & Answer
1

I made some changes to my scraper and wanted to clear the database to store the new data. I used the clear database option from settings and the data seems to be cleared.

But now when I run the scraper it says “Manually ran revision * and completed successfully about a minute ago. nothing changed in the database”

I can’t see any errors in the log.

What am I missing?

Thanks :slight_smile:

LE: it seems there is data in my database, but it just won’t show on me scraper page…

2

I think you are seeing this issue: Nothing saving to database
Its probably just coincidence that you happen to have cleared your DB. You will probably find that your scraper writes a DB once an admin restarts the queue.

3 Likes
3

I’ve done the restart. Have a go and let us know the results in here @aniri and thanks for posting your issue in here :thumbsup:

Thanks for being so helpful as always @chris48s :revolving_hearts:

Luke

4

Thanks Luke, @equivalentideas , it worked! Just one more quick question, if I add more columns to the scraped data, will the db append the new table columns automatically to the data table or not?

2 Likes
5

if I add more columns to the scraped data, will the db append the new table columns automatically to the data table

Yep - as long as you’re using the .save() method it will do that for you: GitHub - openaustralia/scraperwiki-python at morph_defaults

2 Likes
6

Thanks @chris48s :slight_smile: