Newbie google scraping
All,
As a newbie trying out SS BAsic, I'd like to ask for suggestions on how to accomplish the following.
I'd like to create a session that will search Google, and will report a certain site's position agains a selectiong of search phrases.
Is script the right approach for this? I was thinking about storing the list of search phrases in a delimited file, then looping through them, something like this:
a. Main loop that randomly goes through the search phrases
b. If the desired string is found, follow the link, log the page number and the phrase
c. If string is not found, click next
Any suggestions an sample code would be greatly appreciated.
Thanks.
Addict
Hmm... I think your idea
Hmm...
I think your idea sounds good.
The main loop probably could be really simple... something like the script found here: Initialize (Moderate) -- Input from CSV
part 'b' is kind of in your hands, but for logging the page number and phrase to a file, you could use a little script like this:
File file = new File("output/results.txt"); // if you don't specify an 'absolute' path, it will be relative to your screen-scraper folder
out = new FileWriter(file, true);
out.write("data");
out.close();
You'll probably want your script to be more like
out.write(session.getVariable("PAGE_NUMBER") + " : " + session.getVariable("PHRASE"));For iterating pages, I would suggest trying to use the "Next page" script found here: Next Page - Memory Conscious. It can help you iterate search pages in an effecient way.