Database disk image is malformed

1

I’ve got a scraper failing with sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed
https://morph.io/wdiv-scrapers/DC-PollingStations-Dundee
Is there anything that can be done to try and repair the DB?

Thanks,
Chris

2

Any thoughts on this? Maybe if there’s no way to save it, I should clear the DB?

3

Thanks for reporting this @chris48s , sorry about this. I’m not sure of the best steps here, @henare do you have any thoughts?

This seems potentially related to this https://github.com/openaustralia/morph/issues/1051

4

This was almost certainly caused by the disk space problems we were having recently (that are fixed now, yay! :tada:). It’s likely the SQLite scraper database got corrupted during one of these incidents. This should never happen and we’re sorry about that.

I’ve downloaded a copy of the corrupt database and attempted a recovery with some handy steps I found. I’ve then copied this database back up to your scraper @chris48s. Can you please check to see if this is OK?

If it’s not OK then we’ll need to restore your scraper data from backup. Looking at the scraper history we probably need to restore it from the last time the scraper ran successfully, the 13th of December. Please let us know if you’d like us to carry out this restore or if the data, as recovered above, is OK.

1 Like
5

Thanks @henare
That seems to repaired it enough for the scraper to now run, but I seem to have lost quite a lot of data from the history table. Could we could revert to the backup from the last successful run, please.

This was almost certainly caused by the disk space problems we were having recently

Yes. I am aware that SQLite doesn’t respond too well if its storage unexpectedly becoming read-only mid-transaction :slight_smile: The stackoverflow link you posted has brought back some previously repressed memories of trying to recover corrupted SVN repositories after a similar issue.

Cheers

6

I’ve now restored the database from December the 14th, which seems to be the most recent database before the scraper started failing. To do this I ran:

duply sqlite fetch wdiv-scrapers/DC-PollingStations-Dundee/data.sqlite /root/restore/data.sqlite 2016-12-14

then I inspected the file to see if it was OK and replaced the database on the scraper.

@chris48s can you please confirm this is looking good for you?

7

Excellent - thanks very much

2 Likes
8

Hi @henare
I got the same problem with a scraper that I forgot : the DB is now inflated to 6.5Gb so I wanted to save it before I clean it and restart the scraper.
But as I said the DB is now inaccessible,

database disk image is malformed

The project is https://morph.io/bluexm/BTCorderbooks

(Sorry if it’s not the right place to ask for this)
Thanks for any help you can bring