Redirect not working
When I hit a url on browser it redirects me to a new url
Like when I hit www.abc.com it redirects me to www.abc.com/(S(jvtkl4yj4r1d40ei1ogsboeu))/ApplicationSearch.aspx.
The problem is that the parameter (S(jvtkl4yj4r1d40ei1ogsboeu)) changes for every new request made.
I am not able to scrap the site because of this changing parameter.
I have read on Screen Scraper forum that Screen Scraper will automatically follow the redirects but in my case it's not.
In Response Header
(Status-Line) HTTP/1.0 302 Moved Temporarily
and Location /(S(jvtkl4yj4r1d40ei1ogsboeu))/ApplicationSearch.aspx
The above paramter jvtkl4yj4r1d40ei1ogsboeu changes with every new request
Any Solutions for the same.
Screen-scraper will follow
Screen-scraper will follow any HTTP 300 redirects, however, what you're seeing probably a JavaScript redirect. Since screen-scraper doesn't run the JavaScript, it's not followed. In such cases, I always make 2 scrapeable files: the first goes to the page to get the unique ID you need (it's either there or in a separate JS file called from the page), and the second to make the request with the ID set via variables.