can I programmatically click multiple links on same page

Pars,

We're not sure what is causing this error. We have had consistent problems using VBScript with screen-scraper but it's never been fully clear what the causes have been.

One thing we suspect is if you are calling a script from within a script multiple times that it can cause instability. We recommend that you not use VBScript if you have a choice.

If you must the best we can do is wish you luck,

Scott

Pars,

Ok, I think I'm understanding the situation better. We ran into this once before. screen-scraper is always going to URLencode a variable that is used as a POST or GET parameter since that's what [url=http://www.ietf.org/rfc/rfc2396.txt]Tim Berners-Lee[/url] said to do back in 1998.

Within screen-scraper it is unnecessary to include a reference to a bookmark ("a name") in your URL (because it won't affect the server's response), so we don't account for the possibility that one would be included as part of a POST or GET parameter. We assume the entire variable is meant as either a POST or GET parameter (not just a portion) that's why the bookmark portion gets URLencoded along with the rest of the variable.

So, my advice is to extract only the parameter portion of what follows positionID= and try not to worry the bookmark.

If you really, really need/love/want to include the bookmark in the URL you can always bypass the parameters tab and simply construct the URL under the properties tab.

Leave the parameter's tab empty and have this be your URL:

https://ssl.myDomain.com/data/control?action=expand&positionId=~#POSITION_ID#~

screen-scraper will pass this on to the server with the variable untouched.

As for why IE will alter the case and firefox won't, well, um...not sure.

-Scott

Pars,

It's very likely that you will not need to include the # sign in your URL. The only purpose of the # sign in a URL is to force the position of the browser window when the page loads. It's sometimes called a "bookmark" (not to be confused with Favorites/Bookmarks) and makes use of the "name" attribute in an href tag.

http://www.w3schools.com/HTML/html_links.asp

If you omit it from a URL it should not affect the data that is returned to you. So, If you modify your extractor pattern for positionID such that it no longer grabs the # and proceeding text then your current problem should be solved.

However, if you needed to retain the # sign for some reason, you would place my sample code in the script where you're calling the scrapeable file.

Right now you have something like...

session.scrapeFile("My Scrapeable File");

And now you'll have:

I was thinking of a different feature in the other editions of screen-scraper that may have helped with this. You have the option of converting HTML entities at the time you apply an extractor pattern token. Here we're dealing with URL encoding and not HTML entities. My bad.

-Scott

Prs Ethis,

There are some controls in the professional and enterprise editions that may allow you to fix this from within the screen-scraper interface. But for all editions you can utilize the replaceAll() method on the URL string to swap the %23 with a # before setting the value to a session variable to be used as all/part of the URL.

Something like....

Hope that helps,

Scott

Pars Ethis,

The best and most simple approach is to take each request one by one that you've recorded using Live HTTP Headers ([url=http://www.xk72.com/charles/]Charles Proxy[/url] is great, too) and compare it to the request you're making in screen-scraper. Make sure you are not missing anything.

Do this for each request in the processes and resist moving on to the next request until you have the first one working.

-Scott

tia,

Let me suggest a few ideas to you and have you explore them. The first thing you'll probably need to do is create an extractor pattern for all the links on the page that when clicked expand for more content. Once you have this extractor pattern matching for each of the links you'll then create a corresponding script that will handle the request to the server that clicking the link otherwise would do. This usually includes passing post/get parameters to another scrapeable file with, probably in your case, a similar URL to the one with the extracted links. Set your extractor pattern to call this script "After each pattern application".

What this does is creates a recursive loop to call the script and visit the link for every link you've designated on the page. Once it visits all of the links in the matching extractor pattern's dataSet it ends up where it began--at the extractor pattern that began the loop.

Now, while it is visiting each of the links on the page you can have it do additional things like extract data from the expanded content and write that data out to a file. It will do these additional tasks for each of the links and will always return back to where it began when the loop has completed.

Give this a try and post back questions you have them.

-Scott

tia,

It depends on how the additional content is made available on the page. If the content is always there but hidden via CSS then you would need to understand the JavaScript logic that determines, if user clicks here show this data. This might be challenging.

If for each click the data is loaded from the server (even via AJAX) you will need to record the clicks via the screen-scraper proxy and possibly create scrapeable files for each click. Less challenging because the logic is handled by the server.

I hope this helps,

Scott