I made some changes to my scraper and wanted to clear the database to store the new data. I used the clear database option from settings and the data seems to be cleared.
But now when I run the scraper it says “Manually ran revision * and completed successfully about a minute ago. nothing changed in the database”
I can’t see any errors in the log.
What am I missing?
LE: it seems there is data in my database, but it just won’t show on me scraper page…
I think you are seeing this issue: Nothing saving to database
Its probably just coincidence that you happen to have cleared your DB. You will probably find that your scraper writes a DB once an admin restarts the queue.
I’ve done the restart. Have a go and let us know the results in here @aniri and thanks for posting your issue in here
Thanks for being so helpful as always @chris48s
Thanks Luke, @equivalentideas , it worked! Just one more quick question, if I add more columns to the scraped data, will the db append the new table columns automatically to the data table or not?
if I add more columns to the scraped data, will the db append the new table columns automatically to the data table
Yep - as long as you’re using the
.save() method it will do that for you: https://github.com/openaustralia/scraperwiki-python/tree/morph_defaults#saving-data