scrape multiple accounts from same site
I'm scraping a secure site that has a login (my admin), couple of pages of navigation, then a customer account input. This results in a page that I write out to a file and strip out the html table I need for that account. When I do a single account, I get what I need. However, the total time for entire process is approx 20 seconds.
Since I have several thousand accounts, I would like to stay logged in and just input all account ID's in some kind of loop. I need to cut down the time, so was hoping to eliminate the repetitive login/logout per account. Is there a way to do this? The file import seems to be available in enterprise edition only.
Thanks
Pars
scrape multiple accounts from same site
Pars,
I'm guessing you're wanting to loop through the account numbers by reading them in from an outside file. You can do this with the basic edition. You won't actually be importing the file. Rather, you'll be using the file reader class of whatever scripting language you choose.
Here's an example in Interpretive Java.
File inputFile = new File( "MyInputFile.csv" );
FileReader in = new FileReader( inputFile );
BufferedReader buffRead = new BufferedReader( in );
// Read the file in line-by-line.
int index = 0;
while( ( lineFromInput = buffRead.readLine() )!=null)
{
// Check to see if your at the first row
// in case you have a header line.
if (index>0)
{
// Set the current line to a session variable
session.setVariable("myInput", lineFromInput);
// Perform an action here
session.scrapeFile("My Scrapeable File");
}
index++;
}
// Close up the file.
in.close();
buffRead.close();
Your session on the Website will be maintained the entire time unless the server kicks you off. So, you shouldn't need to reauthenticate each time.
Hope this helps.
-Scott