Scrapping a site based on Ajax
Hi,
I have a requirement somewhat similar to "Tutorial:7 Scraping a Site Multiple Times Based on Search Terms". I have been able to follow up what is given in the tutorial.
The site that I am scrapping is will only show 15 results per term. Rest of the results are updated via AJAX when the user scrolls down to the last record. I am not able to scrap these records. I couldn't find anything related to AJAX in the tutorials. Could you point me in the right direction.
Thanks in Advance.
replied via email
Saurabh,
As I mentioned in my email, ites that use AJAX are becoming more and more common. Because of the many different ways a site can make use of AJAX it's not easy to come up with a generic tutorial on the topic. However, here are a few pointers.
- Use screen-scraper's proxy to record the different requests being made via AJAX.
- In needed, make use of the method scrapeableFile.setRequestEntity() for when the site is passing XML in the request payload (this doesn't apply to the site you're scraping).
- Update screen-scraper to the latest alpha release and make use of the feature that allows you to compare a request from a scrapeable file to the request in a transaction from the proxy.
Answers to your specific needs will follow in an email.
-Scott