Just saw the webhook function, if there is a way to program it to send the last record’s date_scraped, it will be handy to monitor any ‘outdated’ scraper
1 Like
I’d be up for adding data to the webhook payload. Unfortunately you wouldn’t be able to do what you describe though.
date_scraped is a concept only known to your scraper, not to morph.io. Furthermore morph.io doesn’t know what the “last record” is because the database is just a database, it doesn’t have last modified metadata on rows.
1 Like
Can we do select max(date_scraped) from data, then output as data and send via webhook?
may work with JSONP callback
Not all scrapers have date_scraped as a database field. What would we do for them?