Scraping all URL's on a site

Hey All,

I have created a SS file that will find and log all files found on a site. THis I will use later on with jMeter to do load testing.

My issue is that this process takes a very long time (30 minutes for 750URLS) (server does 10 req/sec, give or take).

This is the process:

Init -> Scapre homepage -> call write-URL's script -> Load next page ->scrape URL -> call write-URL's script etc etc

The problem is that for each found URL it calls up my "Write URL's" scrip, and as such it's get called very often for all pages found in the menu's.

My second approach was to use runnableScrapingSession.

But then I have two issues:

When called like this:

RunnableScrapingSession = new com.screenscraper.scraper.RunnableScrapingSession( "scrape" );
I get 'session cannot be null'

when called like this:
RunnableScrapingSession = new com.screenscraper.scraper.RunnableScrapingSession( "scrape" );
I get only for professional edition

When using a RunnableScrapingSession it would allow me to just get the URL's and parse them later as a list and I would only start a scraper session once for every page.

In comparison between the two methods

1) 'Standard scraper' would call my script for each and every URL, anf this takes a very long time
2) 'Scripted version' calls a scaper session for each and every unique page, but this doesn't work because I am not using a professional version.

So my question is, what is the recommended approach to just get ALL URL's from a website?

Ries