multi-threading questions

Well, there's a lot to take in and consider here..

Let's start with the multiple threads firing at once:

The workbench isn't really designed to handle multiple instances of the same scrape. You will in fact find some erratic behavior this way. You're idea about the filewriters getting mixed up may also have something to do with it. I wouldn't worry about collision of 2 writes happening at the same time, but if one scrape isn't closing the file fast enough between writes, I could definitely see some weirdness popping up.

Hmm... this is kind of complicated, and the smoothest way I can think to really do what you want is by a mini database. I'm going to explain something that we do here for our own scrapes, and I'll try to connect that to what you're trying to accomplish.

We scrape lots of car, real estate, and insurance sites for clients, and often we have to iterate their sites by zip code. Well, there are 42,000+ zip codes in the US, so that takes a long time. And what if the scrape or computer or server dies only part way through? We have no choice but to restart the scrape at the first zip code. Starting it part way through a list of zip codes takes a file to track it, and requires the scrape to write out its current index in the list.

But we decided to avoid that approach. We wanted to be able to just start up any variable number of scrapes from the server mode (web interface or not, doesn't matter), and we wanted each scrape to simply take the next zip code in our big list. When it was done with that single zip code, it spawns a new dynamic, lazy scraping session (which in turn will dynamically take the next zip in the list all by itself), and then the parent scrape finishes and goes away. So this way we run 42,000 scrapes, each with only one zip. Aside from simple memory advantages, we can choose exactly how many are running, and they could all be running from multiple computers, anywhere, so long as they have access to the list of zip codes.

Best way to do that? A database. MySQL is really simple, and free, so we chose that. This is what we did:

1. PHP file on a web server, accessible to any of the scrapes.
   The PHP just receives a GET/POST parameter "scrape ID" which can simply be the session's name.
   This PHP file connects to the MySQL database, which has a few simple tables, and echoes back the zip code that the scrape should do. It's then up to the scrape to contact this PHP file (scrapeableFile is easiest, since you have extractor patterns at your disposal) and read the response text (the echoed zip code).
2. MySQL tables
   scrapeIDs columns:
   scrapeID VARCHAR(50) PRIMARY KEY
offset INT(11) NOT NULL

   zipcodes columns
   zipcode VARCHAR(5) PRIMARY KEY

   That above PHP simply looks up a query like this:
   SELECT offset FROM scrapeIDs WHERE scrapeID = _utf8'$scrapeName' (where $scrapeName is the GET/POST parameter received)

   From there, the PHP can do another lookup:
   SELECT zipcode FROM zipcodes OFFSET $offset LIMIT 1 (where $offset is the offset retrieved from the first query)

   And there you have the zip code.

   The PHP should then increment that offset value, and save it back into the database:
   UPDATE scrapeIDs SET offset=$newOffset (where $newOffset is the old $offset + 1)

So, even if you don't have any experience in PHP or MySQL, this isn't a tough design. The point is that the "iterator" is kept in the database itself, and is free to iterate over any data that you have in the database. (There should be an extra tiny bit of error handling when you hit the end of the list. Make the PHP restart the offset at 0, or just not echo anything back so that your scrapes stop.)

So for your situation, I think it would be ideal in reply to your comment: I'm assuming I will be able to use an array of RunnableScrapingSessions. This will allow me to reference them with a variable name. Can you suggest a better data type for handling this?

Although it's a completely different approach, it would really simplify this task, especially if you are scraping multiple forums.

You could further simplify our design to suit your own needs: One table called "forumScrapes", which could have one or more entries according to your scrape names. Instead of the "offset" column referring to another database table's offset, you could just use that offset in your scrape itself.

Quick example of some control script used in your scrape:

// This scrapeableFile can easily save the echo'd offset from the PHP into a session variable.
session.scrapeFile("Get next offset");
int start = Integer.parseInt(session.getVariable("OFFSET"))

// since that's what you used in your own examples
int step = 5;

int stop = start + step;
for (int i = start; i < stop; i++)
{
session.scrapeFile("Do the real work");
}

import com.screenscraper.scraper.RunnableScrapingSession;

// Note that I'm not using the "inheriting" version of this constructor. As you mentioned, if you alter one variable in the child, you alter one in the parent-- Not entirely thread safe if you're not expecting it.
nextScrape = RunnableScrapingSession(session.getName());

// Set up any variables you'd like to give to the new scrape.
String[] variables = new String[]{
"var1", "var2", "var3",
};
for (int i = 0; i < variables.length; i++)
nextScrape.setVariable(variables[i], session.getVariable(variables[i]));
nextScrape.scrape();

I went skiing today and really wacked my head pretty good .. having trouble thinking much more than this, currently. Tell me what you think, and if you think you could fit this model to your situation. If you need help with any PHP or MySQL, we do plenty of that as well, so you can ask away. This all really comes down to the difficulty with having a parent scrape keep track of its children. Honestly, you could go down that road, but I think this little database approach is good, since it has the ability to resume at any time, and to run as many of them concurrently as you can (limit imposed by license of the product or by computer power or by bandwidth).

Let me know how this is sounding, and we'll keep working from there.

Tim