urls for java servlets are all identical

Well, I'm sure that your problem can be overcome-- screen-scraper just automates the processes that you either do manually or that the site does behind the scenes. That being said, it could still potentially be tricky in a situation like what you briefly outlined in the original post.

I know that it's been a while now, but can you recall the specific website, or give some specific samples about what was (or was *not*) happening? Without something direct, a reply like mine becomes strict rhetoric about "how screen-scraper works".

Based on what I know of the issue, I would guess that some sort of HTTP transaction is being fired from the applet, and then processed server-side, and then it's replying to the applet, rather than directing your browser to a new page.

If this assumption is correct, then it's probably being handled by one of two things:

1. AJAX-like requests, performed in JavaScript (Asynchronous JavaScript And XML)
2. Requests made directly by an applet, which are completely contained within the applet and thus have no code that shows up in the HTML.

Frankly, in either case, the technique is generally straightforward; the little requests/replies that are made by either AJAX or an applet will show up in the screen-scraper proxy.

AJAX, being XML in nature, it dirt easy to parse, since everything is labeled. A reply to an applet would be similar, I imagine.

The trick is that you're not going to be able to "navigate" an applet very well, as you've discovered. You'll have to analyse the applet and watch your proxy server, and learn how to manually assemble the HTTP requests that the applet is sending out.

For instance, consider a page like the following: http://www.property24.com/search/PropertySearch.aspx?searchtype=Residential . There's a little IFRAME that dynamically reloads its content depending on the various choices you've made in the drop-down boxes.

It would be madness to try to send off the individual HTTP requests (in this case done by JavaScript) for the sake of reconstructing the *exact* process of browsing the page. Instead, watch the proxy server and you'll notice that the IFRAME's contents are coming to the page as seperate entries in the proxy. If you pay attention to the URL of the IFRAME's contents (via the proxy server), you'll see that the numeric code it uses to contruct the URL is simply the various "VALUE" tags from the HTML "<select>" tags in the drop-down boxes.

Since the URL to the data in the IFRAME can be requested independent of the page itself, I could frankly just build the URL to the IFRAME myself, so long as I know the values I want to give it. I could (and *did*, when I scraped this site recently) just proxy the IFRAME upon selecting all the various search options that I'm interested in. Now all I have to change is a single number in the IFRAME's URL to alter the Province that I'm requesting info for. Better yet, I never have to even visit the search page if I have the Province ID numbers from the drop-down box.

Sorry for being long winded, but I hope I'm illustrating the idea. You can often cut some corners while scraping certain sites. In fact, when dealing with pages like the one you've described, you may very well have to cut those corners.

By all means, let me know if you have any questions or concerns!!

Tim (Tim *V*, not Tim *Z*!)