Home » Forums » screen-scraper » screen-scraper support for licensed users

Proxy Session not working

When using my proxy session it errors every time - there is no message, just 'error' in the status field on the progress tab and the page fails to load in a browser. I have tried this on a few http: sites and no luck. I have tried at work and at home in case there are some odd (new) network settings and there are none. The ss program itself works well otherwise?

I would be grateful on any thoughts you may have, as I have a site http://www.machinefinder.uk/ww/en-UK/categories/tractors-by-horsepower-200-plus that I need the proxy for, (it may well be too trick for me in any case....)

Any help on the proxy server and the approach for the site would be gratefully received.

Thanks
Jason

‹ Site rejects due to non browser support Not able to capture South Hams and West Devon council ›
jas777 on 02/01/2016 at 4:07 am

It worked for me using

It worked for me using Firefox. Can you add this to a script, run it, and let me know what the log shows:

log.logScreenScraperInformation();

 jason on 02/01/2016 at 11:11 am

Proxy working on my live server, just not on development Laptop

I have looked at the site I am interested in on my live machine and it has really confused me. I fear that more sites are going to go this way, so I would like to crack it though.

I will look at the proxy server issue on my laptop later....

When I obtain the details of the machinerfinder site through the proxy it gives me a page with the product codes on, which I can scrape. However, when I run the session, the page errors. I assume it needs some kind of variable passing to it somehow, but it doesn't look like a standard POST or GET parameter that it needs. I am wondering if it is a cookie or something.

This creases further issues with the pagination, as there is no next page as such, just a 'view more' where it continues to build on the same page.

http://www.machinefinder.uk/ww/en-UK/categories/tractors-by-horsepower-200-plus with a location filter of United Kingdom is an example of the URL I am interested in. I would like all of these but I don't know where to start to replicate the request that appears in the proxy session and then how to paginate the results. I have hundreds of the 'normal' type of sites being scraped, but this has totally beaten me.

Thanks for your help - I am slowly getting better at this - it has only taken five years!

Jason

 jas777 on 02/04/2016 at 11:47 am