Grabbing Incremental Session Variable Changes

Thanks for your questions! (You're helping us to built a little wealth of knowledge in these forums!)

For your first problem, of repeating the same page over and over again when there are less than 10 pages of results...

I would say that the best way to tell your scrape to bail and move to the next keyword would be to add an extractor pattern to your scrapeableFile which tries to resolve what the final page number for results is. For instance, if I were trying to scrape Google's Image Search, each page clearly tells me several page numbers so that I can jump to whichever I prefer. I could make a pattern for my scrapeableFile which tries to match:


// extractor patterns for the following tokens could be:
// SOME_URL : [^"]*
// SOME_PAGE_NUMBER: \d+

~@SOME_PAGE_NUMBER@~[***now match some thing in the HTML that always comes after the last listed page number***]

For that flagrant last part, about something that appears after the last page number... it could be the "next page" link, it could be a simple "</div", whatever marks the end of the page listings. This way, this pattern will only ever match the last page number listed.

You can then save the "SOME_PAGE_NUMBER" variable, and add it into your page-shifting if statement:
// As quoted from your question, with an addition to the code.
// And... I don't think you meant to write "<= 91" for page comparison :) but I left it anyway.
If session.getVariable( "PAGE" ) == session.getVariable( "SOME_PAGE_NUMBER" ) Then
Call session.setVariable ( "PAGE",NULL )

If session.getVariable( "PAGE" ) <= 91 Then
Call session.ScrapeFile( "Search Results" )
Else
Call session.setVariable ( "PAGE",NULL )
End If

(mind you, I'm no good at VB... I've hardly used it, so I'm only making an educated guess at string-comparing syntax :D )
This way, the session will cut off the current keyword if it hits that "SOME_PAGE_NUMBER" max that it keeps scraping, or it will continue with its keyword so long as the page number is under your desired limit.

As for your second question... I think the easiest thing to do would be to perform that simple calculation. I don't think it would add too much to the processing time. My guess is that you'd want to do something like... (pseudo code with possible Java influence)

currentPage = get the session variable for "PAGE"
currentEntry = dataSet.getNumDataRecords()
keywordEntryNumber = ( currentPage * 10 ) + currentEntry

You could condense that into a single line, if you want, to avoid extra variable assignments. The second line of that code can be used assuming you have a dataSet in scope (which isn't hard... just don't go into nested scripts within scripts, and don't call this calculation "After file is scraped"). All it does is grab the total number of entries made in the dataSet (the cumulative collection of dataRecords during the current execution of the scrapeableFile), which will coincidentally be equal to the number of entries you've found on the current execution of the scrapeableFile.

Useage of the above code might need to be tweaked if you've got some abnormal scope stuff going on. Let me know if this helps you out, or if you encounter more problems with this approach!

Tim