Problem Proceeding Through Site With AJAX
I am trying screen scraper out before I purchase a copy. I am trying to scrape a site that uses ajax. The site uses ajax to progress. I have run a scraping session that has worked when done in the browser but when I try to run this proxy session I get the error "JSMDIRECT Transaction did not complete" after the first page. There seems to be some transactions which are not showing up in the proxy session which are needed to progress in to the site. I notice that the is movement in the progress tab but these disappear. Is there anyway of getting these to put into the scraping session. I know the variables that need to get initalised. A check box is ticked and then it progresses. This site downloads three pages worth of html in the first instance. Then then the ajax allows the other html to shown.
AJAX can be trying, but can
AJAX can be trying, but can usually be done.
If you're not already, try using the latest alpha version of screen-scraper. There is a tool that allows you to save the proxy session for the site, and then when you make a request for a page through screen-scraper, you can go to the scrapeable file > last request tab, and there is a button to compare your screen-scraper request to the original. If you can make the requests exactly the same, you should get the same response.
Sometimes AJAX has custom headers or stuff, but you should have all the tools you need to add those in the scrapeable file API.