Installation Requirements

• Support for Java Runtime Environment of 1.8 or higher

Explanation

The only specific restriction for installing screen-scraper is that your operating system platform supports a Java Runtime Environment of 1.8 or higher. Testing has been done for screen-scraper on Microsoft Windows, Linux, Mac OS X, and other platforms that support a Java Runtime Environment of 1.8 or higher. All systems have managed to run the software without any major changes.

The Windows installer comes with a runtime environment included. Mac OS X and Linux should already have a Java Runtime Environment installed. For help installing screen-scraper on other platforms (e.g., Solaris, FreeBSD) please contact us.

See also:
How much memory and what type of CPU is recommended for screen-scraper?
Scaling & Optimizing screen-scraper

Installation Instructions

Overview

To download screen-scraper, see the page for the edition you'd like to install. Run the installer to set installation options and install screen-scraper. For headless servers, run the installer with a -C flag to indicate the system is headless and the installer shouldn't try to open graphical popups for installation options

• Basic Edition
• Professional Edition
• Enterprise Edition

You may want to compare editions before choosing which to install.

Troubleshooting Linux Installs

screen-scraper ships with a Java Runtime Environment that should work in most distributions of Linux. Because the distributions can vary quite a bit, however, you may need to install a separate Java Runtime Environment and then point screen-scraper to it. If you're able to successfully install screen-scraper, but are having trouble starting it, try downloading the latest Java Runtime Environment from www.java.com for your particular distribution. Once you've installed the JRE, you can point screen-scraper to it by modifying the screen-scraper and server start scripts located in the screen-scraper installation folder.

1. Open the files.
2. Locate the INSTALL4J_JAVA_HOME_OVERRIDE property (near the top of both files).
3. Uncomment the property by removing the pound (#) in front of it.
4. Set the value to the location of the JRE you installed on your system.

As an example: the property might look like this:

This tells screen-scraper to use the JRE located at the given path rather than the one it ships with, or some other JRE that might be on your system.

screen-scraper License Agreement

screen-scraper License Agreement Copyright © 2002-2014 by ekiwi, LLC.
All Rights Reserved.

YOUR AGREEMENT TO THIS LICENSE

After reading this agreement carefully, if you ("Customer") do not agree to all of the terms of this End-User License Agreement ("EULA"), you may not use this Software (hereafter referred to as "Software Product"). Unless you have a different license agreement signed by ekiwi, LLC (hereafter referred to as "ekiwi") that covers this copy of the Software Product, your use of this Software Product indicates your acceptance of this EULA. All updates to the Software Product shall be considered part of the Software Product and subject to the terms of this EULA. Changes to this EULA may accompany updates to the Software Product, in which
case by installing such update Customer accepts the terms of the EULA as changed. The EULA is not otherwise subject to addition, amendment, modification, or exception unless in writing signed by an officer of both Customer and ekiwi. A software license and a license key ("Software Product License"), issued to a designated user only by ekiwi, is required for each concurrent user of the Software Product. By explicitly accepting this EULA you are acknowledging and agreeing to be bound by the following terms:

1. EVALUATION PERIOD

This Software Product may be used in conjunction with a free evaluation Software Product License. You may use the evaluation copy of the Software Product for only thirty (30) days in order to determine whether to purchase the Software Product, after which the Software Product will cease to function. ekiwi bears no liability for any damages resulting from use of the Software Product, and has no duty to provide any support before or after the expiration date of an evaluation license.

2. GRANT OF NON-EXCLUSIVE LICENSE

You may not tamper with, alter, or use the Software Product in a way that disables, circumvents, or otherwise defeats its built-in licensing verification and enforcement capabilities. You may not modify or create derivative copies of the Software Product or this EULA. All rights not expressly granted to you are retained by ekiwi.

ekiwi grants the non-exclusive, non-transferable right for a single user to use this Software Product. Each additional concurrent user of the Software Product must obtain an additional Software Product License. You may install the Software Product on as many computer systems as desired, so long as two copies of the same Software Product License never come into concurrent use.

3. INTELLECTUAL PROPERTY

The Software Product is owned by ekiwi and is protected by international copyright laws and treaties, as well as other intellectual property laws and treaties. You must not remove or alter any copyright notices on any copies of the Software Product. This Software Product copy is licensed, not sold. You may not use, copy, or distribute the Software Product, except as granted by this EULA, without written authorization from ekiwi. ekiwi reserves all intellectual property rights, including copyrights, patents, and trademarks.

4. TRANSFERABILITY

Customer may not rent, lease, lend, or in any way distribute or transfer any rights in this EULA or the Software Product to third parties without ekiwi's written approval, and subject to written agreement by the recipient of the terms of this EULA.

5. PROHIBITION ON REVERSE ENGINEERING AND DECOMPILATION

You may not reverse engineer, decompile, defeat license encryption mechanisms, or disassemble the Software Product or Software Product License except and only to the extent that such activity is expressly permitted by applicable law notwithstanding this limitation.

6. INDEMNIFICATION

You hereby agree to indemnify ekiwi against and hold harmless ekiwi from any claims, lawsuits, liabilty or other losses that arise out of your breach of any provision of this EULA.

7. THIRD PARTY SOFTWARE

Any software provided along with the Software Product that is associated with a separate license agreement is licensed to you under the terms of that license agreement (which license is provided with the Software Product). This license does not apply to those portions of the Software Product.

8. SUPPORT SERVICES

ekiwi may provide you with support services related to the Software Product. Use of any such support services is governed by ekiwi policies and programs described in online documentation and/or other ekiwi-provided materials.

As part of these support services, ekiwi may make available bug lists, planned feature lists, and other supplemental informational materials. ekiwi makes no warranty of any kind for these materials and assumes no liability whatsoever for damages resulting from any use of these materials. Furthermore, you may not use any materials provided in this way to support any claim made against ekiwi.

Any supplemental software code or related materials that ekiwi provides to you as part of the support services, in periodic updates to the Software Product or otherwise, is to be considered part of the Software Product and is subject to the terms and conditions of this EULA.

With respect to any technical information you provide to ekiwi as part of the support services, ekiwi may use such information for its business purposes without restriction, including for product support and development. ekiwi will not use such technical information in a form that personally identifies you without first obtaining your permission.

9. TERMINATION

This EULA terminates on the date of the first occurrence of either of the following events: (1) The expiration of one (1) month from written notice of termination from Customer to ekiwi; or (2) One party materially breaches any terms of this EULA or any terms of any other agreement between Customer and ekiwi, that are either uncorrectable or that the breaching party fails to correct within one (1) month after written notification by the other party.

10. NO WARRANTIES

YOU ACCEPT THE SOFTWARE PRODUCT AND SOFTWARE PRODUCT LICENSE "AS IS," AND EKIWI MAKES NO WARRANTY AS TO ITS USE, PERFORMANCE, OR OTHERWISE. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, EKIWI DISCLAIMS ALL OTHER REPRESENTATIONS, WARRANTIES, AND CONDITIONS, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE, INCLUDING, BUT NOT LIMITED TO, IMPLIED WARRANTIES OR CONDITIONS OF MERCHANTABILITY, SATISFACTORY QUALITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT. THE ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE PRODUCT REMAINS WITH YOU.

11. LIMITATION OF CONSEQUENTIAL DAMAGES

NEITHER EKIWI NOR ANYONE INVOLVED IN THE CREATION, PRODUCTION, OR DELIVERY OF THIS SOFTWARE SHALL BE LIABLE FOR ANY INDIRECT, CONSEQUENTIAL, OR INCIDENTAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE SUCH SOFTWARE EVEN IF EKIWI HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES OR CLAIMS. IN NO EVENT SHALL EKIWI'S LIABILITY FOR ANY DAMAGES EXCEED THE PRICE PAID FOR THE LICENSE TO USE THE SOFTWARE, REGARDLESS OF THE FORM OF CLAIM. EKIWI SHALL IN NO WAY BE HELD LIABLE OR RESPONSIBLE FOR ANY UNLAWFUL OR ILLEGAL USE OF THE SOFTWARE PRODUCT, INCLUDING, BUT NOT LIMITED TO, THE EXTRACTION AND USE OF COPYRIGHTED DATA FROM EXTERNAL SOURCES (E.G. WEB PAGES). THE PERSON USING THE SOFTWARE BEARS ALL RISK AND RESPONSIBILITY AS TO THE USE, QUALITY, AND PERFORMANCE OF THE SOFTWARE.

12. HIGH RISK ACTIVITIES

The Software Product is not fault-tolerant and is not designed, manufactured or intended for use or resale as on-line control equipment in hazardous environments requiring fail-safe performance, including, but not limited to, in the operation of nuclear facilities, aircraft navigation or communication systems, air traffic control, direct life support machines, and weapons systems, in which the failure of the Software Product, or any software, tool, process, or service that was developed using the Software Product, could lead directly to death, personal injury, or severe physical or environmental damage ("High Risk Activities"). Accordingly, ekiwi and its suppliers and licensors specifically disclaim any express or implied warranty of fitness for High Risk Activities. You agree that ekiwi and its suppliers and licensors will not be liable for any claims or damages arising from the use of the Software Product, or any software, tool, process, or service that was developed using the Software Product, in such applications.

13. GENERAL

This EULA is the complete statement of the agreement between the parties on the subject matter, and merges and supersedes all other or prior understandings, purchase orders, agreements and arrangements.

This EULA shall be governed by the laws of the State of Utah. Exclusive jurisdiction and venue for all matters relating to this EULA shall be in courts located in the State of Utah, and you consent to such jurisdiction and venue. If any action is brought by either party to this EULA against the other party regarding the subject matter hereof, the prevailing party shall be entitled to recover, in addition to any other relief granted, reasonable attorney fees and expenses of litigation.

You acknowledge that, in the event of your breach of any of the foregoing provisions, ekiwi will not have an adequate remedy in money or damages. ekiwi shall therefore be entitled to obtain an injunction against such breach from any court of competent jurisdiction immediately upon request. ekiwi's right to obtain injunctive relief shall not limit its right to seek further remedies.

There are no third party beneficiaries of any promises, obligations or representations made by ekiwi, LLC herein. Any waiver by ekiwi, LLC of any violation of this EULA by you shall not constitute or contribute to a waiver of any other or future violation by you of the same provision, or any other provision, of this EULA.

14. CONTACT INFORMATION

If you have any questions about this EULA, or if you want to contact ekiwi for any reason, please direct correspondence to [email protected].

Configure Chrome

Chrome Proxy Settings

Windows users:

1. Open Options under the wrench icon .
2. Click on Under the Hood link on the left-hand navigation.
3. Under Network click on the Change proxy settings... button.
4. Click on LAN Settings
5. Click on the checkbox beginning with Use a proxy server for...
6. Click on the Advanced button.
   
7. In the HTTP and Secure fields type localhost under the Proxy address to use column, and 8777 under Port
   

   If you have changed your proxy server settings to use a port other than 8777 then type your selected port in place of 8777.

8. Hit the OK button a few times till you get back to your web browser.

If you're using a dial-up connection the setup will differ slightly. Instead of the LAN Settings button you'll want to find your dial-up connection under the Dial-up and Virtual Private Network settings dialog box, then configure it via the Settings button.

Depending on your operating system, instead of localhost you may need to use either 127.0.0.1 or the IP address of the machine. If you have trouble connecting to screen-scraper's proxy with your web browser, please see this FAQ.

Linux users:

1. Open Preferences under the wrench icon .
2. Click on Under the Hood link on the left-hand navigation.
3. Under Network click on the Change proxy settings... button.
4. Under the Proxy Configuration tab select the Manual proxy configuration radio button.
   
5. Check the box next to Use the same proxy for all protocols.
6. Enter localhost in the HTTP proxy field.
7. Enter 8777 in the Port field.
8. Click on the Ignored Hosts tab.
9. Enter the following in the Ignore Host List:
• localhost
• 127.0.0.0/8
• *.local
10. Click Close
11. We recommend selecting Close when prompted Do you want to apply these settings system-wide...

If you're using a dial-up connection the setup will differ slightly. Instead of the LAN Settings button you'll want to find your dial-up connection under the Dial-up and Virtual Private Network settings dialog box, then configure it via the Settings button.

Depending on your operating system, instead of localhost you may need to use either 127.0.0.1 or the IP address of the machine. If you have trouble connecting to screen-scraper's proxy with your web browser, please see this FAQ.

Mac OS X users:

1. Open Preferences under the wrench icon .
2. Click on Under the Hood link on the left-hand navigation.
3. Under Network click on the Change proxy settings... button.
4. 
   Select "Manually" next to Configure Proxies
5. Check box next to Web Proxy (HTTP)
6. Enter Localhost under Web Proxy Server and 8777 in adjoining port field
7. Check box next to Secure Web Proxy (HTTPS)
8. Enter Localhost under Web Proxy Server and 8777 in adjoining port field
9. Under Bypass proxy settings for these Hosts & Domains enter localhost; 127.0.0.1

If you're using a dial-up connection the setup will differ slightly. Instead of the LAN Settings button you'll want to find your dial-up connection under the Dial-up and Virtual Private Network settings dialog box, then configure it via the Settings button.

Depending on your operating system, instead of localhost you may need to use either 127.0.0.1 or the IP address of the machine. If you have trouble connecting to screen-scraper's proxy with your web browser, please see this FAQ.

Configuring Firefox

Firefox Proxy Settings

1. Open Options/Preferences window.
   

   Windows users: click Options from the Tools menu.

   Linux users: click Preferences from the Edit menu.

   Mac OS X users: click Preferences from the Firefox menu.

   

2. Click on the Advanced button at the top of the window (if not already selected).
3. Click on the Settings... button at the top of the Network tab.
4. Click the Manual proxy configuration radio button.
   
5. In the HTTP Proxy field type localhost, and 8777 in Port.
   

   If you have changed your proxy server settings to use a port other than 8777 then type your selected port in place of 8777.

6. Click on the Use this proxy server for all protocols check box.
7. Hit the OK button to get back to your web browser.

If you're using a dial-up connection the setup will differ slightly. Instead of the LAN Settings button you'll want to find your dial-up connection under the Dial-up and Virtual Private Network settings dialog box, then configure it via the Settings button.

Depending on your operating system, instead of localhost you may need to use either 127.0.0.1 or the IP address of the machine. If you have trouble connecting to screen-scraper's proxy with your web browser, please see this FAQ.

For useful add-ons, visit the Browser Tools page.

Configuring Internet Explorer

Internet Explorer Proxy Settings

1. Click Internet Options in the Tools menu.
   
2. Go to the Connections tab.
3. Click on LAN Settings.
4. Click on the checkbox beginning with Use a proxy server for....
5. Click on the Advanced... button.
   
6. In the HTTP and Secure fields type localhost under the Proxy address to use column, and 8777 under Port
   

   If you have changed your proxy server settings to use a port other than 8777 then type your selected port in place of 8777.

7. Hit the OK button a few times till you get back to your web browser.

If you're using a dial-up connection the setup will differ slightly. Instead of the LAN Settings button you'll want to find your dial-up connection under the Dial-up and Virtual Private Network settings dialog box, then configure it via the Settings button.

Depending on your operating system, instead of localhost you may need to use either 127.0.0.1 or the IP address of the machine. If you have trouble connecting to screen-scraper's proxy with your web browser, please see this FAQ.

Configuring Opera

Opera Proxy Settings

1. Click Preferences in the Tools menu.
   

   Windows 7 users will need to select Preferences from the Settings menu after clicking to open Opera's menu.

   Mac OS X users will need to select Preferences... from the Opera menu.

2. Click on the Advanced tab.
3. Click on the Network section listed near the bottom on the left.
4. Click on the Proxy Servers... button.
   
5. Enable the first HTTP checkbox, and enter localhost or 127.0.0.1 in the first field and 8777 in the port box.
   

   If you have changed your proxy server settings to use a port other than 8777 then type your selected port in place of 8777.

   

If you're using a dial-up connection the setup will differ slightly. Instead of the LAN Settings button you'll want to find your dial-up connection under the Dial-up and Virtual Private Network settings dialog box, then configure it via the Settings button.

Depending on your operating system, instead of localhost you may need to use either 127.0.0.1 or the IP address of the machine. If you have trouble connecting to screen-scraper's proxy with your web browser, please see this FAQ.

Add a Quick On/Off Button to Toolbar

To simplify the whole process of turning the proxy server on and off, you can add a proxy button to a toolbar. One handy location is on the far right of the tab bar. To add the button,

1. Right-click just about any of Opera's interface toolbars and select Customize....
   

   You might need to select Appearance... explicitly before continuing.

2. Click on the buttons tab.
3. Select preferences to open its options.
4. Drag and drop the Enable Proxy Servers button onto your toolbar.
   

Dragonfly

New to Opera 9.5.x is a feature called Dragonfly. You can access it by selecting Developer Tools from Tools > Advanced, or alternately pressing control-shift-I. If you're familiar with Firefox's Firebug add-on, then you'll quickly recognize Dragonfly. It's built off of the same ideas, allowing you to manipulate CSS in realtime, seeing which properties are being overwritten by another. You can debug Javascript running on the page, find HTML elements on the page by clicking on them so that Dragonfly shows you the corresponding page source, see all final properties of elements, etc. It's a great tool for sifting through a website.

Example Properties File

Overview

The screen-scraper.properties file can be found in the resource/conf directory of screen-scraper's installation. Most available settings and some sample values are listed below.

For the sake of readability the settings are listed in alphabetical order here. In a settings file they are ordered by a serialized value and so not alphabetical.

Example File Contents

Using the Proxy Server

Configure the proxy server

First, create a proxy session to organize your interactions with the specific web sites.

Configure your web browser

Configuring a web browser to use a proxy server is generally pretty straightforward, but varies somewhat for each browser. We have provided instructions on how to setup different browsers:

• Chrome
• Firefox
• Internet Explorer
• Opera

Run the proxy server

Assuming you've configured everything and set up a proxy session, from here you should be able to start up the proxy server by selecting your proxy session in the objects tree and then clicking on the Start Proxy Server button in the general tab. Now just surf the pages that you want to record.

View requests and responses

After you've surfed a bit with your web browser click on the progress tab. From here you can view all of the HTTP and HTTPS requests and responses logged by the proxy server. Clicking on a transaction brings up its details in the lower pane.

Viewing encrypted transactions

If you are using Internet Explorer 7 you have to adjust your security settings. To do this open Internet Options in the Tools menu and under the security tab change the security level to medium.

If security settings are not updated you will see an error page when accessing a site that uses HTTPS encryption.

IE domain mismatch warning

This warning occurs because screen-scraper is using a temporary certificate for encryption that will not match the url that you are accessing. You can safely ignore this warning by clicking Continue to this website. This practice is, however, not recommended.

Most browsers have recently started preventing you from accepting the certificate. As a work-around you can use another proxy service, such as Charles, to proxy SSL sites by installing their root authority certificate on your server (which prevents an error from showing up while using them). You can then import the proxy data into screen-scraper for building your scrape

Using an external proxy server

If you normally use an external proxy server when connecting to the internet (on your local area network, for example), you'll need to specify this information in screen-scraper's external proxy settings. Before you can run the proxy server.

Using Scripts

This feature has been deprecated and by default is not available in the workbench interface. To enable proxy scripting please add AllowProxyScripting=true to your resource/conf/screen-scraper.properties file and restart screen-scraper.

Overview

screen-scraper has the ability to run custom-made scripts while the proxy server is running (more information on starting and stopping the server is available). This allows you to setup blacklists, filter web pages, or otherwise manipulate browser requests and server responses. It is recommended that you read about managing and using scripts before continuing.

Using the scripts

The scripts tab is used to associate scripts with a proxy. Depending on when you decide to run your script, certain built in objects will be in scope that are unique to the proxy environment.

Built-in objects

screen-scraper offers a few objects that you can work with in a script in the proxy environment. See the variable scope section and/or API documentation for more details.

• proxySession: This object allows for interaction with the currently running proxy session.
• request: This object allows for interaction with the currently received HTTP request.
• response: This object allows for interaction with the currently received HTTP response.

Variable scope

Depending on when a script gets run different variables may be in scope (available). The table that follows specifies what variables will be in scope depending on when a given script is run.

Debugging scripts

One of the best ways to fix errors is to simply watch the proxy session log (under the log tab in the proxy session) and the error.log file (located in the log directory of screen-scraper's install directory) for script errors. When a problem arises in executing a script screen-scraper will output a series of error-related statements to the logs. Often a good approach in debugging is to build your script bit by bit, running it frequently to ensure that it runs without errors as you add each piece.

Adding Java Libraries

Overview

screen-scraper allows for Java libraries to be added to the classpath. Simply copy any jar files you'd like to reference into the lib\ext folder found in screen-scraper's directory. The next time you start up screen-scraper it will automatically add the jar files to its classpath. Note that you'll still need to use the import statement within your scripts to refer to specific classes:

screen-scraper was built on a Java 1.5 platform. Your Java scripts must accept at least a version 1.5 JRE in order to compile and run properly.

Anonymization

Overview

Under certain circumstances you may want to anonymize your scraping so that the target site is unable to trace back your IP address. For example, this might be desirable if you're scraping a competitor's site, or if the web site blocks IP addresses that make too many requests.

There are a few different ways to go about this using screen-scraper:

1. Automatic
2. Manual Proxy Pools

If you choose to run anonymous scripts from an external script, it is valuable to read through the documentation on controlling anonymization externally.

Aside from the above methods, you might find our blog posting on how to surf and screen-scrape anonymously helpful. It's slightly dated, but still very relevant.

Mapping Extracted Data

The web interface is only available for enterprise edition users of screen-scraper.

Overview

The mapping tab allows you to alter extracted values. Often once you extract data from a web page you need to put it into a consistent format. For example, you may want products with very similar names to have identical names.

screen-scraper makes use of mapping sets when determining how to map a given extracted value. A mapping set may contain any number of mappings, which screen-scraper will analyze in sequence until it finds a match, or runs out of mappings. As such, you'll often want to put more specific mappings higher in sequence than more general mappings.

Example

Consider the screen-shot of the mapping tab: if the extracted value were Widget 123 screen-scraper would first try to match using the Widget 1 mapping. Because this is an equals match the mapping wouldn't occur, so screen-scraper would proceed to the second mapping. The second mapping would match because a contains type was designated. That is, the text Widget 123 contains the text Widget. As such, the extracted data Widget 123 would become Product ABC, because that is the To value designated for the second mapping.

Using Regular Expressions

When using regular expressions in your mapping you can also make use of back references. Back references allow you to preserve values in the original text when mapped to the To value. For example, if you were mapping the value Widget 123 you could use the regular expression Widget (\d*). In the To column you could then enter the value Product \1, which, when mapped, would convert Widget 123 to Product 123. The value in parentheses in the From column gets inserted via the \1 marker found in the To column.

Running Scraping Sessions within Scraping Sessions

Overview

This feature is only available to Professional and Enterprise editions of screen-scraper.

It is possible to run a scraping session within a scraping session that is already running. This is done with the RunnableScrapingSession class. Detailed documentation on methods available for the RunnableScrapingSession class are available in our API documentation. Here's a specific example of how the RunnableScrapingSession might be used in a screen-scraper script:

Script Overwriting

Overview

Scripts attached to a scraping session are exported along with it. When you subsequently import that scraping session into another instance of screen-scraper it might overwrite existing scripts in that instance. In some cases, though, you might have a series of general scripts shared by many scraping sessions. In these cases you often want to ensure that the very latest versions of these general scripts get retained in a given instance.

Using the Workbench

In the main pane of the script there is a Overwrite this script on import checkbox. When checked, any name clashes between existing scripts and imported versions will prompt you whether it should overwrite the script or not. If the local version's Overwrite this script on import checkbox is unchecked this file will not be overwritten even if you click to have it overwritten.

In a GUI-less environment

Checking the Overwrite this script on import requires access to the screen-scraper workbench, which you may not have access to if screen-scraper is running in a GUI-less environment. In these cases you can make use of the ForceOverwriteScripts property in the resource/conf/screen-scraper.properties file to allow scripts that have this box un-checked to be overwritten. In order to overwrite scripts that have this checkbox un-checked in a GUI-less environment you would follow these steps:

1. Stop screen-scraper (or ensure that it isn't currently running).
2. Open the resource/conf/screen-scraper.properties file.
3. Add this line to the properties file: ForceOverwriteScripts=true (or edit the line, if it already exists).
4. Save and close the properties file.
5. Start up screen-scraper and import the scraping sessions or scripts.

Once you're finished importing you may want to stop screen-scraper, set the property back to false, then start up again. Note that when you import scripts with the ForceOverwriteScripts property set to true screen-scraper will import the scripts regardless of whether or not the Overwrite this script on import checkbox is checked.

Using Extractor Patterns

Overview

Extractor patterns allow you to pinpoint select snippets of data that you want extracted from a web page. It is a block of text (usually HTML) that contains special tokens that will match pieces of data you're interested in extracting. These tokens are text labels surrounded by the delimiters ~@ and @~ (e.g. ~@NAME@~). The identifier between the delimiters can contain only alpha-numeric characters and underscores.

Extractor patterns are added to scrapeable files under the extractor patterns tab.

You can think of an extractor pattern like a stencil. A stencil is an image in cut-out form, often made of thin cardboard. As you place a stencil over a piece of paper, apply paint to it, then remove the stencil, the paint remains only where there were holes in the stencil. Analogously, you can think of placing an extractor pattern over the HTML of a web page. The tokens correspond to the holes where the paint would pass through. After an extractor pattern is applied it reveals only the portions of the web page you'd like to extract.

Extractor tokens designate regions where data elements are to be captured. For example, given the following HTML snippet:

you would extract piece of text by creating an extractor pattern with a token positioned like so:

The extracted text could then be accessed via the identifier EXTRACTED_TEXT.

If you haven't done so already, we'd recommend going through our first tutorial to get a better feel for using extractor patterns.

Tips/Suggestions

• Test your patterns frequently. Extractor patterns take some practice. Especially when you're first trying them out you'll want to test them as you're working with them. It often helps to test it after every couple of tokens you insert.
• Use regular expressions to make your extractor patterns more precise. One of the most common problems encountered occurs when an extractor pattern matches too much data, which usually includes a lot of HTML. There are a couple of ways to address this problem. One is to extend the pattern outward. That is, include HTML that falls before and after the block you're trying to match. The second approach, which is generally the easier of the two, is to include regular expressions. We've included a number of common regular expressions that you can select from the drop-down list. In general, if you can use more precise regular expressions you can reduce the amount of HTML in the extractor pattern. Doing so makes your patterns more resilient to changes that might be made to the web site you're scraping.
  

  If an extractor pattern takes too long to match a block of text it will timeout. The timeout setting may be adjusted from the general tab of the Settings located in the Options menu. If you find that your extractor pattern is timing out you might try adjusting it by using more precise regular expressions.

• Ensure that the pattern extracts the number of data records you expect it to. Oftentimes your pattern might not be as flexible as you think it is. Test it out to make sure it matches as many times as you think it should.
• Try tidying the HTML. This will ensure that white space is handled consistently and will often clean up extraneous characters. The setting that determines whether or not HTML gets tidied is adjusted under the advanced tab of the scrapeable file.

Using Scripts

Overview

screen-scraper's scraping engine allows you to associate custom scripts with various events in the scraping process. It is recommended that you read about managing and scripting in screen-scraper before continuing.

Using the scripts

Depending on what event triggers a script to be run different objects will be in-scope. Triggers regarding the scraping session are added on the general tab of the scraping session, file request/response triggers are associated on the properties tab of the scrapeable file, and extractor pattern events in the scripts section of the main tab in the extractor patterns tab of the scrapeable file.

Scripts can also be used to run scripts using the session.executeScript method.

Built-in objects

screen-scraper offers a few objects that you can work with in a script in the scraping engine. See the variable scope section and/or API documentation for more details.

• session: The running scraping session.
• scrapeableFile: The file interaction including request and response, it also holds the extractor pattern requests.
• dataSet: All of the matches from an extractor pattern's tokens.
• dataRecord: A single match of an extractor pattern's tokens.

Variable scope

Depending on when a script gets run different variables may be in or out of scope. When associating a script with an object, such as a scraping session or scrapeable file, you're asked to specify when the script is to be run. The table that follows specifies what variables will be in scope depending on when a given script is run. Only variables that are in scope are accessible to the script.

Debugging scripts

One of the best ways to fix errors is to simply watch the scraping session log and the error.log file (located in the log directory where screen-scraper was installed) for script errors. When a problem arises in executing a script screen-scraper will output a series of error-related statements to the logs. Often a good approach in debugging is to build your script bit by bit, running it frequently to ensure that it runs without errors as you add each piece.

When screen-scraper is running as a server it will automatically generate individual log files in the log directory for each running scraping session (this can be disabled in the settings window). An error.log file will also be generated in that same directory when internal screen-scraper errors occur.

The breakpoint window can also be invaluable in debugging scripts. You can invoke it by inserting the line session.breakpoint() into your script.

Using Session Variables

Overview

Session variables allow you store values that will persist across the life of a scraping session.

Setting session variables

There are a few different ways to set session variables.

1. Within a script using the setVariable method of the session object
2. Designate that the value matched by a extractor token should be saved in a session variable (check Save in session variable in the main tab of the extractor token).
3. Using the RemoteScrapingSession object from external sources (such as a PHP or ASP script) via their setVariable methods (see scripting in screen-scraper for more details).

Retrieving values from session variables

As with setting session variables, there is more than one way to retrieve values of session variables.

1. Within a script using the getVariable method of the session object.
2. Embed the identifier for the session variable, surrounded by ~# and #~ delimiters.
   

   If you have a session variable identified by QUERY_PARAM you might embed it into the URL field of a scrapeable file using http://www.mydomain.com/myscript.php?query=~#QUERY_PARAM#~. screen-scraper will automatically replace the ~#QUERY_PARAM#~ with the value of the session variable.

Using Sub-Extractor Patterns

Overview

Sub-extractor patterns allow you to extract data in the context of an extractor pattern, providing significantly more flexibility in pinpointing the specific pieces you're after. Consider a search results page consisting of rows and columns of data. Using normal extractor patterns you would use a single pattern to extract the data from all columns for a single row. In many cases this works just fine; however, the process gets more complicated when each row differs significantly. For example, certain cell rows may be in different colors or their contents may be completely missing. With a normal extractor pattern it would be difficult to account for the variability in the cells. By using sub-extractor patterns you could create a normal extractor pattern to extract an entire row, then use individual sub-extractor patterns to pull out the individual cells.

When using sub-extractor patterns only the first match will be used. That is, even if a sub-extractor pattern could match multiple times, only the data corresponding to the first match will be extracted. Because of this sub-extractor patterns are not always the correct method for getting data within a larger context. To get multiple matches in a larger context, like all rows in a table, you would instead use manual extractor patterns.

Example

Consider the following HTML table:

Here is the corresponding HTML source:

It would be difficult to write a single extractor pattern that would extract the information for each row because the contents of the cells differ so significantly. The different colored cells and the cell spanning two columns make the data too inconsistent to be easily extracted using a single pattern (which would require lots of regular expressions and might still prove impossible or inconsistent).

Consider this extractor pattern:

The ~@DATARECORD@~ extractor pattern token is special in that it defines the block of data to which you wish to apply sub-extractor patterns. Sub-extractor patterns cannot be applied to a token with a name other than DATARECORD

If applied to the HTML above the extractor pattern would produce the following three matches:

Sub-extractor patterns would allow you to extract individual pieces of information from each row. For example, consider this sub-extractor pattern:

If applied to each of the individual extracted rows above the following three pieces of information would be extracted:

This is a simple case. Now consider the extractor pattern for the phone number:

If applied to each of the individual extracted rows above the following three pieces of information would be extracted:

In the case of Sherry Lloyd this presents a serious problem because she does have a phone number listed. It is not selected because of the additional class. Let's adjust the sub-extractor pattern slightly:

The ~@nondoublequotes@~ represents an extractor token that uses the Non-double quotes regular expression: [^"]*. Matching anything between where it is covering until it encounters double quotes. In this particular case Sherry's phone number also gets extracted.

We now have the case of the cell in the second row that spans two columns, which would not get extracted by our current sub-extractor patterns. We may still want this information, however, so we create the following sub-extractor pattern, just in case the cell exists:

If applied to our data we'd get the following results:

When multiple sub-extractor patterns hold a token with the same name (in this case, PHONE), the last one to match is the one that determines the value of the token. In this example either one or the other will match. If both could match then we would want to have the first phone extractor pattern ordered later than the one to match the no-data-available pattern

Sub-extractor patterns aggregate everything that's extracted into a single data set. Using all of our extractor and sub-extractor patterns together we'd get the following data set:

Important Notes

• When two sub-extractor patterns hold a token with the same name, the one that doesn't match anything will have no effect. Sub-extractor patterns are applied in sequence, and those that match something will take precedence over those that don't.
• ~@DATARECORD@~ is the extractor token identifier that defines the block of data to which you wish to apply sub-extractor patterns. You cannot use sub-extractor patterns without using this token name in the main extractor pattern.
• When using sub-extractor patterns only the first match will be used. That is, even if it could match multiple times, only the data corresponding to the first match will be extracted.

Introduction

To ensure clarity, the first thing that you need to know about the workbench is the names for the various regions of the window. This will help to keep us oriented correctly during the documentation of the workbench.

Workbench Layout

• Menu Bar: Options available at the very top of the page (just below the title bar) starting with File on the far left.
• Buttons Bar: Just below the File Menu this area contains buttons for such common tasks as saving, creating new objects, cut, copy, and paste.
• Objects Tree: Located on the left side of the window, this pane lists all of the screen-scraper objects that are currently available to this instillation of screen-scraper.
  

  If this is your first time opening screen-scraper the only item listed will be the root folder.

• Main Pane: This is the largest pane in the workbench and usually takes up at least two-thirds of the window. It changes to reflect the content of the object that is selected in the assets pane. Because of this feature, it is most commonly referred to by the name of the screen being displayed in it.
• Status Bar: Strip at the bottom of the window that reports on the current memory usage of screen-scraper. Occasional messages will also show up, such as when screen-scraper has saved your work.

The size of the two panes can be adjusted to your liking by clicking on the vertical bar that divides the two panes and dragging it to the left or right.

Settings

Overview

This section contains a description of each of the screens found in the Settings window, which can be displayed by selecting Settings from the Options menu, or by clicking the wrench icon in the button bar.

Proxy Sessions

Overview

A proxy session in screen-scraper is a record of the requests and responses that go between a browser and a proxy server. It is useful in learning how to scrape a site and is used to configure screen-scraper's proxy server. For more information see our documentation about using the proxy server.

Managing Proxy Sessions

Adding

1. Select New Proxy Session from the File menu.
2. Click on the globe in the button bar.
3. Right click on a folder in teh objects tree and select New Proxy Session.
4. Use the keyboard shortcut Ctrl-J

Removing

• Press the Delete key when it is selected in the objects tree
• Right-click on the proxy session in the objects tree and select Delete.
• Click the Delete button in the proxy session general tab.

Scraping Sessions

Overview

A scraping session is simply a way to collect together files that you want scraped. Typically you'll create a scraping session for each site from which you want to scrape information.

Managing Scraping Sessions

Adding

• Select New Scraping Session from the File menu.
• Click the gear in the button bar.
• Right click on a folder in the objects tree and select New Scraping Session.
• Use the keyboard shortcut Ctrl-K

Removing

• Press the Delete key when it is selected in the objects tree
• Right-click on the scraping session in the objects tree and select Delete.
• Click the Delete button in the general tab of the scraping session.

Importing

• Right-click on the folder in the objects tree that you want to import the files into (other than the root folder) and select Import Into. In the window that opens, navigate to and select the scraping session you want to import.
• Select Import from the File menu. In the window that opens, navigate to and select the scraping session you want to import.
• Add the scraping session to the import folder in screen-scraper's install directory
  

  screen-scraper should not be running when you add the file into the folder. All files will be imported into the root folder the next time screen-scraper starts.

Exporting

When a scraping session is exported it will use the character set indicated under the advanced tab. If a value isn't indicated there it will use the character set indicated in the general settings.

• Right-click on the scraping session in the objects tree and select Export.
• Click the Export button in the general tab of the scraping session.

Scrapeable Files

Overview

A scrapeable file is a URL-accessible file that you want to have retrieved as part of a scraping session. These files are the core of screen-scraping as they determine what files will be available to extract data from.

In addition to working with files on remote servers, screen-scraper can also handle files on local file systems. For example, the following is a valid path to designate in the URL field: C:\wwwroot\myweb\my_file.htm.

Managing Scrapeable Files

Adding

• Click the Add Scrapeable File button on the general tab of the desired scraping session.
• Right click on the desired scraping session in the objects tree and select Add Scrapeable File.

Removing

• Press the Delete key when it is selected in the objects tree
• Right-click on the desired scrapeable file and select Delete.
• Click the Delete button in the properties tab of the scrapeable file.

Extractor Patterns

Overview

Extractor patterns allow you to pinpoint snippets of data that you want extracted from a web page. They are made up of text (usually HTML), extractor tokens, and possibly even session variables. The text and session variables give context to the tokens that represent the data that you want to extract from the page.

Extractor patterns can be difficult to understand at first. We recommend that you read about using extractor patterns or go through our first tutorial before continuing.

Managing Extractor Patterns

When creating extractor patterns you should use the HTML that will be found under the last response tab associated with a scrapeable file. By default, screen-scraper will tidy the HTML once it's been scraped, meaning that it will format it in a consistent way that makes it easier to work with. If you use the HTML by viewing the source for a page in your web browser it will likely be different from the HTML that screen-scraper generates.

Adding

• Click the Add Extractor Pattern button in the extractor patterns tab of the scrapeable file
• Select desired text in the last response tab of the scrapeable file, right click and select Generate extractor pattern from selected text.

Removing

• Click the Delete on the desired extractor pattern.

Sub-Extractor Patterns

Overview

Sub-extractor patterns allow you to extract data within the context of an extractor pattern, providing significantly more flexibility in pinpointing the specific pieces you're after. Please read our documentation on using sub-extractor patterns before deciding to use them.

Sub-extractor patterns only match the first element they can. To get multiple matches, you would use manual extractor patterns instead.

Managing Sub-Extractor Patterns

Adding

• Click the Add Sub-Extractor Pattern button in the sub-extractor patterns tab of the extractor pattern.

Removing

• Click the Delete Sub-Extractor Pattern button in the main pane of the sub-extractor pattern in the sub-extractor patterns tab of the extractor pattern.

Extractor Tokens

Overview

Extractor tokens select the information from a file that you want to be able to access. The purpose of an extractor pattern is to give context to the extractor token(s) that it contains. This is to assist in getting the tokens to only return the information that you desire to have. Without extractor tokens you will not gather any information from the site.

Extractor tokens become available to dataRecord, dataSet, and session objects depending on their settings and the scope of the scripts invoked. All extractor tokens are surrounded by the delimiters ~@ and @~ (one for each side of the token). Between the two delimiters is where the name/identifier of the token is specified.

Managing Extractor Tokens

Adding

• Type ~@TOKEN_NAME@~ in the appropriate location in the Pattern text of the extractor or sub-extractor pattern.
  

  Make appropriate changes to the TOKEN_NAME text to reflect the desired name of the token.

• Select a portion of the extractor pattern, right click, and select Generate extractor pattern token from selected text

Removing

• Remove the token and delimiters from the Pattern text of the extractor pattern like you would with any text editor

Editing

• Double-click on the desired extractor token's name
• Select the extractor token's name, right click, and choose Edit token

Scripts

Overview

screen-scraper has a built-in scripting engine to facilitate dynamically scraping sites and working with data once it's been extracted. Scripts can be helpful for such things as interacting with databases and dynamically determining which files get scraped at when.

Invoking scripts in screen-scraper is similar to other programming languages in that they're tied to events. Just as you might designate a block of code to be run when a button is clicked in Visual Basic, in screen-scraper you might run a script after an HTML file has been downloaded or data has been extracted from a page. For more information see our documentation on scripting triggers.

Depending on your preferences, there are a number of languages that scripts can be written in. You can learn more in the scripting in screen-scraper section of the documentation.

If you haven't done so already, we'd highly recommend taking some time to go through our tutorials in order to get more familiar with how scripts are used.

Managing Scripts

Adding

• Select New Script from the File menu.
• Click on the pencil and paper icon in the button bar.
• Right click on a folder in the objects tree and select New Script.
• Use the keyboard shortcut Ctrl-L.

Removing

• Press the Delete key when it is selected in the objects tree.
• Right-click on the script in the objects tree and select Delete.
• Click the Delete button in the main pane of the script.

Importing

• Right-click on the folder in the objects tree that you want to import the script into (other than the root folder) and select Import Into. In the window that opens, navigate to and select the script you want to import.
• Select Import from the File menu. In the window that opens, navigate to and select the script you want to import.
• Add the script to the import folder in screen-scraper's install directory
  

  If screen-scraper is running when you copy the files into the import folder they will be imported and hot-swapped in the next time a scraping session is invoked. They will also be imported if you start or stop screen-scraper.

Exporting

• Right-click on the script in the objects tree and select Export.
• Click the Export button in main pane of the script.

Other Windows

Overview

So far we have explained each of the windows in the workbench of screen-scraper. Here we would like to make you aware of a few other windows that you will likely come across in your work with screen-scraper.

File Path Delimiters

Overview

Because screen-scraper internally uses Java it is important that file paths follow the requirements of Java. That is that file paths follow the Unix/Linux structure (e.g., /usr/local/file.txt). If you are working on a machine that follows these conventions then it will not look any different to you; however, if you are working on a Windows machine this is an important difference to keep in mind.

Windows uses the backslash (\) as a file delimiter but Java uses it as an escape character. That means that on a Windows machine you need to pay closer attention to file paths as they will look a little different.

Windows file paths should use either a forward slash (/) or two backslashes (\\) to delimit file paths.

Windows Example

Scripting in Interpreted Java

Overview

screen-scraper uses the BeanShell library to allow for scripting in Java. If you've done some programming in C or JavaScript you'll probably find BeanShell's syntax familiar. Documentation for BeanShell is excellent, and we'd recommend referring to it as you program.

Interpreted Java is just a phrase used to mean that it is java that does not require being compiled.

See the using scripts and API pages for details on objects and methods that you can make use of in a script. We also use Interpreted Java in all of our tutorials, which should get you familiar with how it's used in screen-scraper.

It is possible to access Java libraries in screen-scraper. See adding Java libraries for more details.

Example

Java Tutorials

We use Java in the screen-scraper tutorials but if you would like to learn more about Java you can look for tutorials online. The following are some good Java resources:

• jalbum
• Oracle Java Tutorials

Scripting in JScript

This scripting language is not available by default any more. To use it you will need to edit the AllowUnstableWindowsFeatures in the screen-scraper.properties file.

Overview

Writing scripts in JScript gives you the familiarity of a widely used language, while still providing access to commonly useed Windows libraries. Using JScript within screen-scraper can only be done on a Windows platform, and requires that the JScript runtime be installed. The chances are good that you've already got the JScript runtime on your system.

screen-scraper will automatically detect if the JScript runtime is installed, which you can see by selecting a script from the objects tree in the workbench and clicking on the Language drop-down list. If you don't see JScript in the list then the runtime needs to be installed.

If you do not have JScript runtime on your system you can download it from Microsoft's script downloads page.

Please be aware that because of a bug in the third-party library that allows screen-scraper to integrate with the Microsoft Scripting Engine problems can occur if multiple JScript scripts are run simultaneously. If you're using the professional edition of screen-scraper and plan on running multiple scraping sessions simultaneously you should use Interpreted Java, JavaScript, or Python as a scripting language.

Available Objects

Because screen-scraper uses the native JScript engine, all Active X objects installed on the computer (such as ADO or the FileSystemObject) can be accessed. Additionally, all of the objects mentioned on the Using scripts and API pages are also available.

Example

Java classes can also be instantiated within a script using the CreateBean function. For example, the following script will instantiate a RunnableScrapingSession and run it:

Scripting in JavaScript

Overview

Mozilla's Rhino scripting engine is used by screen-scraper to allow scripts to be written in JavaScript. Documentation for Rhino is sparse, but the interpreter does adhere strictly to the established ECMAScript standard, so just about any reference on JavaScript could be referred to. If you try writing scripts using JavaScript, and run into difficulties (because of lack of documentation), you may want to consider using Interpreted Java instead, which has very similar syntax and provides significantly better documentation. If you've worked with client-side JavaScript in web programming, you'll probably be comfortable using JavaScript in screen-scraper.

Examples

Classes in standard Java Library

Packages outside standard Java Library

These must be prefaced with the Packages keyword.

Scripting in Perl

This scripting language is not available by default any more. To use it you will need to edit the AllowUnstableWindowsFeatures in the screen-scraper.properties file.

Overview

screen-scraper uses ActiveState's ActivePerl library for scripts written in Perl. Using Perl within screen-scraper can only be done on a Windows platform, and requires that the ActivePerl runtime be installed.

screen-scraper will automatically detect if the ActivePerl runtime is installed, which you can see by selecting a script from objects tree in the workbench and clicking on the Language drop-down. If you don't see Perl in the list then the runtime needs to be installed.

The ActivePerl runtime can be downloaded from ActiveState's download page for free.

Example

Java classes can be instantiated within a script using the CreateBean function. For example, the following script will instantiate a RunnableScrapingSession for the "Weather" scraping session (which is found in the default screen-scraper installation) and run it:

Scripting in Python

Overview

The Jython interpreter is used by screen-scraper to for scripting in Python. Jython is a very fast interpreter, and we'd recommend using it if you're familiar with the Python programming language.

Importing External Java Classes

Importing your externally-compiled classes is as easy as placing them in the ./lib/ext folder of your installation. The Jython interpreter will automatically include that folder on your PythonPath.

Generator Objects

The generator objects are implemented in Jython and the folders lib/ext, lib/jython-lib, and lib/jython-lib/site-packages are included in python's system path.

Example

When scripting in Python all of the standard Java classes can be used. Classes must be imported using the Java package hierarchy of screen-scraper, which is also required if you'd like to create one of screen-scraper's RunnableScrapingSession objects. Here's an example that will run a scraping session called "Weather":

Notice that before the RunnableScrapingSession class can be used it first must be imported.

Scripting in VBScript

This scripting language is not available by default any more. To use it you will need to edit the AllowUnstableWindowsFeatures in the screen-scraper.properties file.

Overview

If you've programmed in Visual Basic or Active Server Pages you should find scripting in screen-scraper to be similar. Using VBScript within screen-scraper can only be done on a Windows platform, and requires that the VBScript runtime be installed. The chances are good that you've already got the VBScript runtime on your system.

screen-scraper will automatically detect if the VBScript runtime is installed, which you can see by selecting a script from the objects tree in the workbench and clicking on the Language drop-down list. If you don't see VBScript in the list then the runtime needs to be installed.

If you do not have VBScript runtime on your system you can download it from Microsoft's script downloads page.

Please be aware that because of a bug in the third-party library that allows screen-scraper to integrate with the Microsoft Scripting Engine problems can occur if multiple VBScript scripts are run simultaneously. If you're using the professional edition of screen-scraper and plan on running multiple scraping sessions simultaneously you should use Interpreted Java, JavaScript, or Python as a scripting language.

Available Objects

Because screen-scraper uses the native VBScript engine, all Active X objects installed on the computer (such as ADO or the FileSystemObject) can be accessed. Additionally, all of the objects mentioned on the using scripts and API pages are also available.

Example

Java classes can also be instantiated within a script using the CreateBean function. For example, the following script will instantiate a RunnableScrapingSession and run it:

Web Interface: Settings

Overview

The web interface settings can be opened by clicking on the settings button in the upper-right corner of the screen.

Settings

• Timeout: The default number of minutes the scraping session is allowed to run before a request to stop is inserted.
  

  If this value is blank, 0, or negative, the scraping session will not time out.

• Time: The default percentage of time whereby two runs of a scraping session may differ without being flagged as a possible error.
• Record Count: The default percentage of records scraped whereby two runs of a scraping session may differ without being flagged as a possible error.
• Repeat Every: How often the scrape should be rerun by default.
• Reload From File: If you directly edit the screen-scraper.properties file this causes the new settings to be reloaded.
• Save: Save settings and close dialog.
• Cancel: Close without saving changes to settings.

Flagged scrapes are highlighted in red in the run/running tab.

Web Interface: Runnable tab

Overview

This tab displays all scraping sessions loaded into the current instance of screen-scraper. It will display basic information on scraping sessions that are currently running, as well as scraping sessions that have run in the past. It also allows you to start and schedule scraping sessions.

The runnable tab will display all of the scraping sessions listed alphabetically by name, and the messages from the most recently started instance of the scrape.

• View as: Change from list to folders view using this drop-down menu.
• Refresh: Update the contents of the scraping session list.
• (List of available scraping sessions):
  • Name: The name of the scraping session.
  • Start Time: The date and time the scraping session was last started.
  • Running Time: The amount of time the scraping session has been running (the number will update each time you click the Refresh button at the top right of the table).
    

    If the scraping session is not currently running it shows is how long is took to run last time it was run.

  • Previous Running Time: The amount of time the scraping session took the last time it ran.
    

    If the scraping session is not currently running it show the amount of time it took to run two times ago.

  • Num Records: The number of records the scraping session has extracted as recorded by the session.addToNumRecordsScraped method. If the method is never called then this number will always be zero.
  • Previous Num Records: The number of records the scraping scraping session the last time it ran.
  • Status: Indicates the current status of the scraping session. Possibilities include "In Process", "Completed", "Interrupted", and "Error".
  • Export: Exports the scraping session, just as you would from the workbench.
  • Run Now: Runs the scraping session.
  • Schedule: Allows you to schedule the scraping session to be run. See schedule scraping sessions for more information.
  • Remove: Deletes the scraping session from screen-scraper.
  • Notes: Allows you to view the notes specified in the scraping session.

Web Interface: Run/Running tab

Overview

This tab displays information on scraping sessions that are either currently running or have run in the past. You can use this table to compare run times, the number of records scraped, and also to monitor scraping session logs. If scraping sessions have timed out (see settings) the stop button will gray and the status will change to interupted. If a script has flagged a fatal error (see setFatalErrorOccurred) then the error cell will display in red for that scrape.

Scrapes can be ordered in ascending and descending order using any of the fields. This is done by clicking on the column header that you want to sort by.

Run/Running tab

• Stop Marked Scraping Sessions: Stops the scraping sessions whose rows are checked on the far left.
• Remove Completed Scraping Sessions: Removes the scraping sessions which have a status of complete.
  

  Removing records for scraping sessions that have run doesn't remove the scraping sessions themselves, just the records related to the time when they were run.

• Remove Marked Scraping Sessions: Removes the scraping sessions whose rows are checked on the far left from the run/runnable tab.
  

  Removing records for scraping sessions that have run doesn't remove the scraping sessions themselves, just the records related to the time when they were run.

• Auto-refresh: Refreshes the table of running files regularly.
• Refresh: Refreshes the table of running files.
• (List of running and completed scraping session runs):
  • Name: The name of the scraping session.
  • Start Time: The date and time the scraping session was last started.
  • Running Time: The amount of time the scraping session has been running (the number will update each time you click the Refresh button at the top right of the table).
    

    If the scraping session is not currently running it shows is how long is took to run last time it was run.

  • Previous Running Time: The amount of time the scraping session took the last time it ran.
    

    If the scraping session is not currently running it show the amount of time it took to run two times ago.

  • Num Records: The number of records the scraping session has extracted as recorded by the session.addToNumRecordsScraped method. If the method is never called then this number will always be zero.
  • Previous Num Records: The number of records the scraping scraping session the last time it ran.
  • Status: Indicates the current status of the scraping session. Possibilities include "In Process", "Completed", "Interrupted", and "Error".
  • Error: Indicates whether or not a fatal error has been flagged in the scraping session (see setFatalErrorOccurred).
  • Error Message: In the event of a flagged error, displays the provided message (see setErrorMessage).
  • Peek: Pops up a box that allows you to view the most recent section of the log.
  • Stop: Stops the scraping session.

Web Interface: Scheduled tab

Overview

On this tab you can manage scraping sessions that have been scheduled to be run. The columns can be sorted by clicking on the column headers.

Scheduled Tab

• Refresh: Reloads the table of scheduled scraping sessions.
• (List of scheduled runs for scraping sessions):
  • Scraping Session: The name of the scheduled scraping session.
  • Timeout: The amount of time in minutes the scraping session should be allowed to run.
    

    If this value is 0 or a negative number, the scraping session will not time out.

  • Date/Time: The date and time the scraping session is next scheduled to be run.
  • Session Variables: Any session variables that are to be passed to the scraping session when it runs.
  • Disable/Enable: Allows you to temporarily enable or disable the scheduled run of the scraping session.
    

    If the run of the scraping session is disabled, it will not run even if it's scheduled to do so.

  • Edit: Pops up a dialog box that allows you to manage the scheduled run of the scraping session.
  • Remove: Removes the scheduled run of the scraping session.

Web Interface: Schedule Scraping Session

Overview

It can be very helpful to have scraping sessions run automatically or on an on going basis. The web interface makes this simple allowing you to schedule and manage multiple scrapes in a single location.

Managing Scheduled Scrapes

Scheduling Run

• Click the Schedule button on the runnable tab.

Editing Scheduled Run

• You can alter the settings for an already scheduled scraping session by clicking on teh Edit button on the scheduled tab.

Removing Scheduled Run

• You can remove an already scheduled scraping session by clicking on the Remove button on the scheduled tab.

Scraping Engine API

Overview

The scraping engine is the backbone of screen-scraper and provides four built-in objects. These objects are: session, scrapeableFile, dataSet, and dataRecord. We have also included the RunnableScrapingSession class as it best pertains to the engine.

For details on which objects are available to scripts in the context of a scrape see the variable scope section of the documentation.

Objects

• dataRecord: This gives access to the most recently extracted data record. This will most likely only be used in scripts that get accessed after each time an extractor pattern is applied. This object simply extends Hashtable, and documentation on the Hashtable's methods can be found in Java's documentation.
  

  The dataRecord object is populated using the names of tokens from extractor patterns.

• dataSet: The dataSet object holds all data records extracted by an extractor pattern after it has been applied as many times as possible to the HTML retrieved by a scrapeable file. A data set is analogous to a result or record set that would be returned from a database query. A data set contains any number of data records, which are analogous to rows in a database.
• log: Methods used for logging information.
• RunnableScrapingSession (com.screenscraper.scraper.RunnableScrapingSession): This is a class that can be instantiated within a script in order to run a scraping session. The Maximum number of concurrent running scraping sessions in the settings dialog box will control how many scraping sessions can be run simultaneously.
• scrapeableFile: This refers to the scrapeable file that is currently being requested and analyzed.
• session: This variable refers to the currently running scraping session.
• sutil: General methods for checking and manipulating data.

DataRecord
--- A_FLOAT : 3.14159
--- A_LIST : List
------ Element 0 : Value 1
------ Element 1 : Value 2
------ Element 2 : Value 3
------ Element 3 : Set
--------- Element : A value
--------- Element : More value
--------- Element : Other stuff
--- A_MAP : Map
------ KEY_1 : 1
------ KEY_2 : 2
------ KEY_3 : 3
--- A_SET : Set Logged above as "------ Element 3 : "
--- A_STRING : Screen-Scraper
--- AN_INT : 5