I’ve got a scraper failing with sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed https://morph.io/wdiv-scrapers/DC-PollingStations-Dundee
Is there anything that can be done to try and repair the DB?
This was almost certainly caused by the disk space problems we were having recently (that are fixed now, yay! ). It’s likely the SQLite scraper database got corrupted during one of these incidents. This should never happen and we’re sorry about that.
If it’s not OK then we’ll need to restore your scraper data from backup. Looking at the scraper history we probably need to restore it from the last time the scraper ran successfully, the 13th of December. Please let us know if you’d like us to carry out this restore or if the data, as recovered above, is OK.
Thanks @henare
That seems to repaired it enough for the scraper to now run, but I seem to have lost quite a lot of data from the history table. Could we could revert to the backup from the last successful run, please.
This was almost certainly caused by the disk space problems we were having recently
Yes. I am aware that SQLite doesn’t respond too well if its storage unexpectedly becoming read-only mid-transaction The stackoverflow link you posted has brought back some previously repressed memories of trying to recover corrupted SVN repositories after a similar issue.
I’ve now restored the database from December the 14th, which seems to be the most recent database before the scraper started failing. To do this I ran:
Hi @henare
I got the same problem with a scraper that I forgot : the DB is now inflated to 6.5Gb so I wanted to save it before I clean it and restart the scraper.
But as I said the DB is now inaccessible,