Extracting data from tables with varying row counts

Hello! The solution to the problem is exactly why we built screen-scraper :)

Quick rundown: Extractor patterns search over the page and find as many matches as they can. Thus, a single pattern could match 600 times on a single page if it can find that many.

The HTML is formatted so that each table row begins with a TR tag, and it just so happens that they all have that valign="TOP", which helps to identify those rows you're after.

There are two ways you could solve this obstacle, and they're both about the same. Consequently, we'll only worry about the simpler one:

Since the pattern is going to match multiple times all by itself, you may as well just make the pattern consist of a single row:
~@DATETIME@~ ~@CITY@~ ~@STATE@~ ~@SHAPE@~ ~@DURATION@~ ~@SUMMARY@~ ~@POSTED@~

• junk_parameters: [^>]* (one of the default entries in the dropdown box "HTML tag parameters")
• junk_some_url: [^"]* (default entry "Non double-quote")
• DATETIME, POSTED: [^<>]* (Default entry "Non HTML tags")
• CITY, STATE, SHAPE, DURATION, SUMMARY: Don't put anything in the regex textbox for these tokens' patterns. Sometimes the page is missing data for a cell, like for a state or shape. Leaving it blank will allow the pattern to simply match whatever's between the starting point of the pattern and the very next "</td>" tag.

Now it will match as many times as it can (which should be equal to the number of rows on the page).

From there, you can make 2 little scripts; one to get rid of extra HTML in your data (right now, the empty cells actually contain a "<br />") and a second one to write out your data to a CSV (basic tab-delimited spreadsheet format).

Script number one; this one gets rid of those silly "<br />" things. Set this one to run after your pattern, and make sure the "When to run" box says "After each pattern application".
// Set the language of the script using the drop-down/combo box
// just above the big text field for the script; set it to JAVA.
//
// You could literally just copy and paste this script out of this reply and paste it into screen-scraper.
//
// Here we go...!

for (Enumeration e = dataRecord.keys() ; e.hasMoreElements() ;)
{ varName = e.nextElement();
dataRecord.put(varName, dataRecord.get(varName).replaceAll("<[^>]*>", ""));
}

Aaaaand, script number 2. Set this one to run after the pattern, and make sure the "When to run" box says "After pattern is applied":
// Set the language of the script using the drop-down/combo box
// just above the big text field for the script; set it to JAVA.
//
// You could literally just copy and paste this script out of this reply and paste it into screen-scraper.
//
// Here we go...!

dataSet.writeToFile( "C:/site_data/extracted_data.csv" );

In that second one, you can change the last line to point to whatever folder you want, and screen-scraper will put your spreadsheet output there. (Don't use backslashes between folder names, use forwardslashes like in the example.)

As a final note, you should know that the second script is simply going to dump all of your extractor pattern variables into the CSV file. Consequently, our two junk variables, "junk_parameters" and "junk_some_url" are going to get their own columns in the CSV, too. Using this quick method of writing out to a CSV file, you'll have to manually get rid of those columns since you probably don't want them :)

If you need help with anything more, be sure to let me know!

Tim