script needed for this?
So, if all I want to do is scrape each page of a search, do I need a script to do that or do I just need to create a scrapeable file for each page?
I'm not a programmer so some of this stuff is a bit over my head. Thanks for any info you can provide.
--sam
script needed for this?
OK, so I went through the tutorials and I've run into a snag. My script seems to loop through the first page 99 times instead of looping through each iteration. I'm sure I've bungled something, but what exactly I don't know...
I have two scripts that I modified from the tutorials or from snippets posted here on the boards... Here's my page looping script...
{
runnableScrapingSession = new com.screenscraper.scraper.RunnableScrapingSession( "DCI Database" );
runnableScrapingSession.setVariable("PAGE", String.valueOf(i));
runnableScrapingSession.scrape();
}
the variable "PAGE" has been added to the URL in the Scrapable File so that the URL looks like this...
http://webapp.wizards.com/ratings.aspx?Page=~#PAGE#~&BrandRatedSeasons=26&ratingseasonid=26&Demographic=1%7C1&action=SearchAnd then I added this modified script to generate a file...
try
{
session.log( "Writing data to a file." );
// Open up the file to be appended to.
out = new FileWriter( "Epic.txt", true );
// Write out the data to the file.
out.write( dataRecord.get( "Ranking" ) + "\t" );
out.write( dataRecord.get( "Link" ) + "\t" );
out.write( dataRecord.get( "Name" ) + "\t" );
out.write( dataRecord.get( "Rating" ) + "\t" );
out.write( dataRecord.get( "Location" ) );
out.write( "\n" );
// Close up the file.
out.close();
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}
The file is generated just fine. However, it's the same page 1 results over and over. I'm sure my lack of understanding when it comes to scripts is to blame here. Any help is greatly appreciated.
--sam
script needed for this?
Thanks for the info. Just one last question. Does it make more sense to create a script if I have a lot of search pages to go through? I don't mind learning how to script if it will make my life easier in the long run... just wondering if it's worthwhile if I have like 99+ search pages to scrape.
If scripting is the most efficient way of doing things, I'm sure I'll have questions about that once I delve into it a bit. All I do now is SQL and a tiny bit of Excel macros.
My test at creating a scrapeable file were successful so I'm really excited about using this tool. Nice job, Developers! :)
--sam