Scraping Google SERPS - Is it supposed to be harder than other websites?
Just wanted to quickly ask if Google has some way of scrambling it's html output so that screen scraping is difficult. I've been having trouble reliably matching patterns. I've spent a good few hours trying to figure out a solution but I thought I should ask someone if I'm being over ambitious.
I'm ready to give up, but I hate giving up!
Many Thanks,
Sebastian
I've scraped them before, and
I've scraped them before, and they are littered with AJAX, which makes them harder, but not impossible. They aren't trying to obfuscate the results, but it sure looks like they are.
503 error also a problem
Thanks for getting back to me Jason.
I am now stumbling across 503 (please prove you're human) errors where it seems Google has cottoned on to my automated queries. This is the final nail in the coffin for me.
Incidentally, I was trying to build a table of stats for keyword lists contained within a CSV file where the final output is 4 columns: Keyword, Intitle Count, Inanchor Count, Exact Match Count. These stats are obtained from google via a series of queries like intitle:"My Keyword". I was pursuing this was because I thought that the competition analysis data would come in very handy when doing keyword research for SEO.
I'd like to post my scraping session here so that other (more competent) users might be able to make use of the time I've put into this (if I could figure out how to mask my IP with a proxy and match the simple pattern "About XXX Results" I think it would work), but I can't see any option for leaving attachments in this forum. If anyone would like me to email them a copy, please message me by putting my username for this forum before @yahoo.com
Cheers
Attaching the File
If you would like to attach a file it has to be attached to the main post. Comments don't allow for file attachments but the forum topic does. You should have an edit button that will allow you to edit your initial question and an option for attaching a file in there.
Just if you want to attach it instead of having people contact you.
Can't See an Option for Attaching a File
Your suggestion would be preferable Tylers, but either I'm blind, stupid, or both, but I can see any option for attaching a file even when I click the edit tab on the original post or when attempting to create a bran new forum topic post.
Appreciate the help though.