Nothing saving to database

Question & Answer
1

Using scraperwiki.sqlite.save isn’t saving anything to the database on any of my scrapers.

Is anyone else having this issue?

2

I think it’s a very big problem with sqlite - allmost all scrapers can’t write to dbase
Look at almost any scraper in Explore section - they all have “Nothing changed in database”

3

Yes. It certainly looks that way. I think they need to turn it off and on again. :wink:

1 Like
4

Same here… Is anyone having success with alternative save methods?

Steve

5

It seems affect ALL scrapers.
Hey, devs Where are you?

1 Like
6

I’ve tried running a test scraper locally and it produces these weird chars in data.sqlite. Don’t know if they should be there or not, I’m not an expert on sqlite.

data.sqlite:

b�b@aindexdata_adataCREATE UNIQUE INDEX data_a ON `data` (`a`)Z�tabledatadataCREATE TABLE `data` (
            `a` integer
��        , `bbb` text)
       	Hi there
��
7

I’m taking a look at this issue now. I’ll let you know what I find

8

Hey @pietrobenito, @achasik, @lowndsy

Things look to be working again at least on a test scraper https://morph.io/morph-test-scrapers/test-ruby. Could you let me know if things are all working for the scrapers you’re looking at?

Thanks,
Matthew

1 Like
9

Its working now - Thank you!

1 Like
10

@mlandauer Thanks!

It’s working again now.

Do you know what happened?

11

Thanks, looks good for me too!

12

OMG database not writeable again
Just created new scraper in node.js from template and ran it
"Nothing changed in database"
https://morph.io/achasik/test

My old scraper affected too

13

Looks to be working again now.

14

It happens periodicaly. Now again nothing changed in database. Can you fix it?

15

I’ve just created a bunch of scrapers and they’ve run, reported success but no data has been created to download. I think I might be hitting the same problem…

Here’s a couple of examples:
morph.io/wdiv-scrapers/DC-PollingStations-Eden
morph.io/wdiv-scrapers/DC-PollingStations-Halton

16

@chris48s, @achasik sorry this is happening. I haven’t figured out the root cause of this problem yet but what fixes it in the short term is restarting the background queue processes.

17

Again the same problem

18

I’m guessing they’ve recently migrated to RabbitMQ and it keeps crashing.

19

Thanks for reporting this @achasik. @mlandauer is away so, for my sins, I’m now looking after morph.io until he returns in October.

I’ve now restarted the queue, as Matthew advised we should do when this happens, and it’s working again. He told me that this issue was still happening every few days so I’ve been trying to keep an eye out for it but please post in here if you see it happen again.

Thanks again for looking out for this!

20

My scraper was stuck in the queue for a day, so I just stopped it and re-ran - nothing is saving again, sorry!