Unable to scrape a details page - could it be cookie-related?
Hi
I am trying to scrape these products:
smvcommercials.co.uk/used-vehicles/page-1/default/-1/-1.aspx
This page returns fine and I can grab the product page URLS. The problem is that when SS attempts to read these pages they are blank. I have run a proxy transaction and there are some differences in the cookies and headers, but I am not sure what is important and how to set them (never encountered this issue before)
Thanks
Jason
Jason, On first glance, that
Jason,
On first glance, that is a dynamic page. When you request a details page there is a bunch of responses, some of which are JSON objects. Have you dealt with those before? They can be picky about the HTTPHeaders, or it could be a cookie, but I think the previous more likely.