Get help with morph.io and scraping

Webhook POST data

Just saw the webhook function, if there is a way to program it to send the last record’s date_scraped, it will be handy to monitor any ‘outdated’ scraper

1 Like

I’d be up for adding data to the webhook payload. Unfortunately you wouldn’t be able to do what you describe though.

date_scraped is a concept only known to your scraper, not to morph.io. Furthermore morph.io doesn’t know what the “last record” is because the database is just a database, it doesn’t have last modified metadata on rows.

1 Like

Can we do select max(date_scraped) from data, then output as data and send via webhook?

may work with JSONP callback

Not all scrapers have date_scraped as a database field. What would we do for them?