Note: this questions has been copied from the screen-scraper FAQ to this forum.
"I'd like to insert the information I'm extracting into a database. Can screen-scraper do that for me?"
"The current version of screen-scraper won't automatically insert information into a database for you, though you have a number of options for getting the data into a database. In a future version screen-scraper will automate much of this process for you, but for now here are the primary options you have:
* In a screen-scraper script write the information out to a text file, then, after all of the information has been scraped, write code in the language of your choice to read the information from the file and insert it into a database.
* Write objects that will take the information extracted by screen-scraper and insert the information into a database. For example, if you're using VBScript to write scripts in screen-scraper you might write a DLL or two in Visual Basic that will take the data and insert it into the database. If you prefer Java, you could write Java classes that will do the same thing.
* Write scripts in screen-scraper that will insert the information directly into the database. This would involve using technologies like ADO with VBScript or JDBC with Interpreted Java to issue the SQL statements directly to the database.
* Run screen-scraper as a server, invoke it from an external application, retrieve the scraped information, then have your external application insert the information into a database. As complex as that might sound, this is actually the most common (and recommended) approach. For exmaple, you might consider running screen-scraper as a server and invoking it from a Visual Basic application. The Visual Basic application would run a scraping session which store the extracted data in session variables. Once the scraping session is finished the Visual Basic application would request the value of the session variable, which would return the full data set of extracted information. The Visual Basic application could then iterate through the data set, inserting or updating information in a database.
As a side note, it is by design that screen-scraper doesn't insert information automatically into a database for you. The appraoch we've taken to the design of screen-scraper is to ensure that it does one thing very well: extract information from web sites. Generally related to that process, however, are subsequent steps that involve manipulating and cleaning up the information, as well as storing it in some persistent mechanism (such as a database or text file). All of those things can be done by screen-scraper, but we've designed screen-scraper primarily to handle data extraction. "
Web Data Extraction
Can screen-scraper insert into databases?
"The current version of screen-scraper won't automatically insert information into a database for you, though you have a number of options for getting the data into a database. In a future version screen-scraper will automate much of this process for you, but for now here are the primary options you have:
* In a screen-scraper script write the information out to a text file, then, after all of the information has been scraped, write code in the language of your choice to read the information from the file and insert it into a database.
* Write objects that will take the information extracted by screen-scraper and insert the information into a database. For example, if you're using VBScript to write scripts in screen-scraper you might write a DLL or two in Visual Basic that will take the data and insert it into the database. If you prefer Java, you could write Java classes that will do the same thing.
* Write scripts in screen-scraper that will insert the information directly into the database. This would involve using technologies like ADO with VBScript or JDBC with Interpreted Java to issue the SQL statements directly to the database.
* Run screen-scraper as a server, invoke it from an external application, retrieve the scraped information, then have your external application insert the information into a database. As complex as that might sound, this is actually the most common (and recommended) approach. For exmaple, you might consider running screen-scraper as a server and invoking it from a Visual Basic application. The Visual Basic application would run a scraping session which store the extracted data in session variables. Once the scraping session is finished the Visual Basic application would request the value of the session variable, which would return the full data set of extracted information. The Visual Basic application could then iterate through the data set, inserting or updating information in a database.
As a side note, it is by design that screen-scraper doesn't insert information automatically into a database for you. The appraoch we've taken to the design of screen-scraper is to ensure that it does one thing very well: extract information from web sites. Generally related to that process, however, are subsequent steps that involve manipulating and cleaning up the information, as well as storing it in some persistent mechanism (such as a database or text file). All of those things can be done by screen-scraper, but we've designed screen-scraper primarily to handle data extraction. "