Issue : Download file generated at run time
Screen scraper application is unable to download file generated at run-time. I tried to download file using downloadFile method but it is downloading file which contains HTML text instead of expecting data.
Steps to download file
----------------------
1. Login to web site.
2. Go to file download page
3. Click on file download link (Hyperlink re-directing to someURL/index.php?G1=value&G2=value&G=value) - This PHP script process the requested file and then provides option to download file)
For this one application in particular:
* We navigate to the login page and pass POST data to the login handler/form - all captured by the proxy.
* We then navigate to the "download files here" link, or pass the absolute URL to get the desired file directly - proxied navigation.
The proxy handles all of this beautifully, and shows the appropriate pages captured during the session when I examine it's results.
Issue
------
The problem is that I don't understand or can't get the logic inserted in the scraping session to grab the required file. I have tried several methods, but always end up with a file containing the HTML of the logon page--I have no idea how that is happening.
That's a sticky one but it
That's a sticky one but it can be done.
While the site is generating your file, you will get sent to a page that says something like "please wait", and it will refresh every few seconds via meta tags or JavaScript. You will need to emulate that refresh.
So say you make a request to http://www.foo.com?picID=bar you will get an HTML response. Your scrape detects that, waits 10 seconds, and makes the same request again. Again detect if it's your image and repeat if needed until your response is the desired image.
There are some variations in implementation sometimes, but that's the gist of it.