A comparison site using screen scraper. How?

bcb,

There a different ways you could go about creating a comparison site. How much you involve screen-scraper depends on how much of the processing you would like done by other applications involved.

Because screen-scraper's primary role is that of a data aggregator you will need additional code to start up screen-scraper, send it the search terms and handle the results that screen-scraper returns.

You can control screen-scraper from a number of different languages and technologies but let's say your website is running in PHP and you have a MySQL database where you store your data. From PHP you can start up a scraping session and send it the search terms.

Now, you have the option to send your search terms to a single scraping session which will then dispatch those terms to other scraping sessions. These other scraping sessions will crawl each site that you are wanting data from.

While scraping you can have screen-scraper write the scraped data to your MySQL database. Then your PHP app can query the database for the data and display it on your website.

If time is critical and you want the data presented on your website as quickly as possible you can use an alternative approach. You can have PHP dispatch the search terms to each scraping session directly then use a technique that will allow your PHP app to receive the scrapped data in real-time as it is being scraped.

That is the basic up-shot of how screen-scraper can be used to set up a comparison site. Unfortunately, we don't have any working samples we can share.

We have a fully-operational scrapbooking comparison website (scrapbookfinds.com) which we've maintained for a number of years. It uses screen-scraper and a few open source software applications. We have also created a number of comparison sites for clients but they are proprietary and we could not share the code with others.

We, of course, offer to do the work for you. If you're interested in a free quote just drop us a line.

Thanks,
Scott