1: Tutorial Details

How it Works

There are a number of ways to insert scraped data into a database, of the ones outlined in this FAQ we will be using the last one: POST's extracted data to a local web-enabled script that accepts the data and inserts it into a database.

The Enterprise edition of screen-scraper has the ability to handle scraped data in real time. As of right now, this has been implemented in the Java and PHP drivers for screen-scraper. If you're running the Enterprise Edition, and want to interact with screen-scraper using either of those languages, read over the Handling Scraped Data in Real Time section of either our Invoking screen-scraper from Java or Invoking screen-scraper from PHP pages for details on this. This tutorial doesn't cover this approach, but it's quite a bit easier and cleaner to implement than the method that will be described here.

The basic idea in this tutorial is that we'll have a special scrapeable file that will POST data to a PHP file, which will handle inserting the data into a database. The flow of events will look like this:

1. Extract the data from the site, saving each value (product title, price, etc.) in session variables.
2. Invoke a Save product scrapeable file, which will POST the extracted data to a PHP file.
3. The PHP file accepts and validates the data. If the data was incomplete it will return an error message and then attempt to insert the data into the database itself. If everything went well it will return a success message.
4. In screen-scraper we use an extractor pattern to check the success/failure status of the error message.