Get help with morph.io and scraping

API atom feed is no longer generating correctly

The API atom feed is no longer generating correctly, the feed appears empty.

This has been the case since around 1st July 2019, impacting all my scrapers.

I’ve raised https://github.com/openaustralia/morph/issues/1224 to track this.

The URL I’m using for testing is:

https://api.morph.io/Kooben209/OT-PP/data.atom?key=XXXXXXXXXXXX&query=select%20address%20||%20price%20as%20content%2C%20title%20as%20title%2C%20link%20as%20link%2C%20pubdate%20as%20date%20from%20"otmdata"%20limit%2010

Could you check that and confirm that it looks like something you’d expect to work? In particular, I’d like to check what values you’re passing for content/title/date/link

1 Like

Hey @Kooben209, this is all fixed now. You found a pretty obscure bug! it was to do with the parsing of the date format in the pubDate field in your database when it’s used as the time field in the atom feed. Because of the specific format of the date-time you used it broke the conversion but it’s all fixed now so hopefully everything should be working fine now.

All the best,
Matthew

2 Likes

Thanks very much for quick response. I manged to fix it myself by using DATE(pubDate) in the sql query.

Great, thanks for the fix. I see that I do not need to do the date conversion now and can use the original format which includes the time component.