Scripting a .NET site with JavaScript

Question & Answer
1

Hi all

I am trying to scrape http://eservice.ryde.nsw.gov.au/DATracking/Modules/ApplicationMaster/default.aspx?page=found&1=thismonth&4a=DA&6=F and to select another pages within the A tag are done by JavaScript, anyone have example code for PHP where I can use to change to Page 2, 3 etc?

My set of source code is here on https://github.com/LoveMyData/city_of_ryde and it is already working on morph.io but I want to get multi pages scraping working.

Any assistance is appreicated.

Cheers
Eric

PS: I am a hobbyist programmer, ideally, copy and paste set of code is the best if anyone in this community have it.

1 Like
2

After studying the Javascript, it was a simple POST request and all sorted.

1 Like
3

Bravo! :tada: :thumbsup:

4

I just wanted to add to this a little because it’s a common problem when scraping gov’t sites.

Here’s an example where I’ve used post requests to deal with paging on one of these ugly sites. It uses NodeJS.

https://morph.io/drzax/morph-division-postcode-correspondence

The tricky was parsing the state data out of the form and the mimicking the behavour of __doPostBack function before sending the request. In my case, it also seemed to be necessary to send additional HTTP headers to make the server behave as expected. I’m not sure which ones were ultimately necessary, but I just copied the headers from Chrome dev tool and added them to each request.

1 Like