Looping a scraping session?

Hi Joe,

I think a different approach may be in order. screen-scraper really isn't designed to handle passing data between scraping sessions (as evidenced by your experience). As such, I think the best approach would be to invoke screen-scraper from an external application written in something like Visual Basic or Java. Ideally, you should do as little programming in screen-scraper as possible. If you use an external application you'll have a much easier time debugging.

To get you started, I'd recommend taking a look at our second tutorial (http//www.screen-scraper.com/support/tutorials/tutorial2/tutorial_overview.php), which gives a basic example of this.

Best,

Todd

Hi,

I'll add this as a bug for us to investigate. In the meantime, if you insert a slight delay in between each of the sessions does it make a difference?

Best,

Todd

Setting the maximum number of concurrent sessions to 1 should fix this. Did it not?

Todd

Hi Joe,

In answer to your questions:

1. Are you wanting to cycle through a list of names, setting each name as a session variable for each scrape? If so, I'd probably do something like this:

import java.util.ArrayList;

names = new ArrayList();
names.add( "Joe" );
names.add( "Sally" );
names.add( "Brent" );

for( int i = 0; i < names.size(); i++ )
{
runnableScrapingSession = new com.screenscraper.scraper.RunnableScrapingSession( "LoginSession" );

runnableScrapingSession.setVariable( "ACCOUNT", names.get( i ) );

runnableScrapingSession.scrape( "LoginSession" );
}

You can find documentation on the ArrayList class here: http://java.sun.com/j2se/1.5.0/docs/api/java/util/ArrayList.html.

2. This code would generate the random number:

import java.util.Random;

generator = new Random();

myRandomNumber = generator.nextInt( 27 );

3. Hm. Not that I can think of. I think you'd need to use an array. I supposed you could do something tricky along the lines of converting an integer into an ASCII character by its code. You might Google around a bit. Someone's probably written code for this.

4.
a) It depends on the operating system. If you're using Windows the Task Scheduler does a good job. On a Unix variant (e.g., Linux) you'll probably want to use cron.
b) There are a number of ways you could do this. If you're invoking screen-scraper from the command line you'll probably just need to write out files with the various values. If you're writing an application to invoke screen-scraper (e.g., Java or Visual Basic), that application could handle tracking which values have been set when. That may not sound like a very good answer to the question, but it really just depends on what your preferences are and what programming languages you're comfortable with.

And in answer to your last posting:

There are a number of ways you could do this. This is just one off the top of my head:

for( int i = 1; i < 1000; i++ )
{
num = "00" + String.valueOf( i );
num = num.substring( num.length() - 3 );
}

Kind regards,

Todd

Also I am extremely annoyed with myself. I am not a programmer and I can't even figure out how to pause.

How do you pause while their is not a scraping session running yet?

How do you pause within a session?

Where can i find out these commands.

I looked all over the web for an answer for hours.

For a 1 second pause I tried

Pause.Pause (1000);
Session.Pause (1000);
Pause (1000);
com.screenscraper.scraper.Pause (1000);

I am so annoyed. I wasted 2 hours trying to pause for one second!

I looked at beanshell.org... I tried googling java help. I tried all kinds of stuff. I can't find good help for easy easy easy stuff.