Add lithgow scraper to planningalerts-scaper

1

Hi

I sent an email (contact@planningalerts.org.au) to add the lithgow scraper to the planningalerts-scraper but haven’t had a response and doesn’t look like the data has been imported.

Could someone add it when they get a chance? The url is https://morph.io/adamclayton/lithgow_city_council_development_applications

Thanks

Ed

2

Hi

Any progress on getting the scraper and data merged with planningalerts? I have a friend in Lithgow who was bugging about it …

Also if there is any issue with the way I implemented the scraper for planningalerts please let me know.

Thanks

Ed

3

Thanks for the contribution Ed, there were a few errors during the auto run, do you know what are they and can they be minimise?

can you update date_received field to ISO-8601 format please?

look like you are also missing ‘date_scraped’ field as required field.

Here is the reference https://www.planningalerts.org.au/how_to_write_a_scraper on the standard and required fields

4

Thanks for the response. The error could be related to the ASP form pages. I will need to check this a bit more as I thought I had fixed it.

I will look into the other suggestions and update you when this is done

Thanks

Ed

5

Checking the scraper now and it seems like it is hanging.

The other errors from the previous run also show that the scraper was running for up to one day before erroring out.

Other than that the rest of the time it is ok.

Any reason why the scraper is just sitting there doing nothing before timing out?

6

The scraper gave an error code:
Last run failed about 9 hours ago with status code 137.

The logs show nothing. I’m wondering if the error code is because the scraper was timing out?

Anyway I have suspended it for now and will put in a handler to deal with the timeout.

7

Thanks, let us know once you have resolve those issues and we can put it to the PlanningAlerts site.

8

Been running as scheduled with no issues. The date format is also fixed

Thanks

9

Thanks for this great contribution @adamclayton :hammer_and_wrench: :champagne: I’ve forked it to https://morph.io/planningalerts-scrapers/lithgow_city_council_development_applications and I’m setting up the new authority in PlanningAlerts.

Thanks for guiding this along @LoveMyData :heart:

10

This is now loaded into PlanningAlerts https://www.planningalerts.org.au/authorities/lithgow Thanks @adamclayton and @LoveMyData :revolving_hearts: