Concurrent sessions
Hi
I am new to SS and using the following script to extract prices that are calculated on a pricing site.
It works fine and takes about 12-15 seconds to run through one record.
I have up to 50.000 records and therefore I need to speed up the process.
Can you help me make my script scrape the pages in parallel with for example 10 concurrent scrapes?
Thanks
Sverre
import java.io.*;
////////////////////////////////////////////
session.setVariable("INPUT_FILE", "input.csv");
session.setVariable("OUTPUT_FILE", "output.csv");
////////////////////////////////////////////
// Creates outputfile
FileWriter out = null;
out = new FileWriter( session.getVariable("OUTPUT_FILE"), true );
BufferedReader buffer = new BufferedReader(new FileReader(session.getVariable("INPUT_FILE")));
String line = "";
// Read from input file
while (( line = buffer.readLine()) != null ){
String[] lineParts = line.split(";");
// Sets session variables to values from input file
session.setVariable("VAR_1", lineParts[0]);
session.setVariable("VAR_2", lineParts[1]);
.
.
.
session.setVariable("VAR_68", lineParts[68]);
//Scrape pages with input values VAR_1,...,VAR_68
session.scrapeFile("PAGE_1");
session.scrapeFile("PAGE_2");
session.scrapeFile("PAGE_3");
//Save scraped values in ouput file
try
{
// Open up the file to be appended to.
out = new FileWriter( session.getVariable("OUTPUT_FILE"), true );
// Write out the data to the file.
out.write(session.getVariable("VAR_1") + ";");
out.write(session.getVariable("VAR_2") + ";");
.
.
.
out.write(session.getVariable("VAR_100") + ";");
out.close();
}
}
// Close inputfile
buffer.close();
You would need the enterprise
You would need the enterprise edition to get up to 10. I would make one scrape that you start, and it's job is to read in the CSV input, and then launch a separate scrape that will go get the data for each line in the CSV. You would do that with http://community.screen-scraper.com/API/Runnable_scraping_session
Hi Jason Thanks - it worked
Hi Jason
Thanks - it worked just perfect.
Sverre