Home » Forums » screen-scraper » screen-scraper public support

Tips for how to approach scraping this site?

I'm trying to scrape this page: http://ipldata.msdlouky.org/IPLCISEntry.aspx

I have a list of property addresses in column "A" of an excel spreadsheet. Ideally, i'd like to use this spreadsheet to populate the fields on this site. However, this website requires the street number and street name be input seperately so I could, if needed, export the data into into columns "A" and "B" of an XLS spreadsheet for the street number and street name to make it easier.

There are posts about importing from files but they only include instructions for the first column only. Is it easy to grab just the street number and street name separately to populate form data?

To give you an idea of the manual version of what i'm trying to do, I:

1) visit this site: http://ipldata.msdlouky.org/IPLCISEntry.aspx
2) Input the property NUMBER into the "From" and "Number" fields. (Example: for 2200 Main St., you'd put "2200" in both of these fields).
3) Then in the "Street Name" you would put something like "Main". The other fields aren't required.
4) After pressing "Search", it comes up with a result. If there isn't a result, i need that sent to an output file.
5) If there is a result, you have to click on the number in the "application key" column to bring up the details, and I want to scrape this data and output it to a spreadsheet.

It seems fairly simple but i don't know where to start. Can anyone you offer some suggestions of templates I might be able to use?

What would be the best way to approach scraping this site?

‹ Outputting UTF-16 text files from SS? Pausing random periods inbetween each pattern match ›
jeffcole on 06/11/2011 at 3:57 pm

It's tons easier to use a CSV

  1. It's tons easier to use a CSV file than Excel, so would recommend saving your sheet as a CSV
  2. You will need to parse out the number from the street name. You can do it on the fly while the scrape reads the CSV or you can do as you say and put 2 columns in your input--the latter might be a cleaner solution so my example uses this method.

So you would start the scrape with a script like this, and for each row in the CSV it would request the page:

import java.util.*;
import au.com.bytecode.opencsv.*;

// Indicate the location and name of the input file
fileName = "input/addresses.csv";
File inputFile = new File(fileName);

// Read the file
if (inputFile.exists())
{
        // Start the CSV reader
        CSVReader reader = new CSVReader(new FileReader(fileName));
       
        // Read all rows of the CSV
        rows = reader.readAll();

        // Iterate rows
        int index = 0;
        for (i=0; i<rows.size(); i++)
        {
                // Skip headers
                if (index>0)
                {
                        // Parse line
                        line = rows.get(i);
                        num = line[0];
                        street = line[1];
                       
                        session.log("Requesting " + num + " " + street);
                       
                        // Save variables
                        session.setv("NUM", num);
                        session.setv("STREET", street);
                       
                        // Scrape file
                        session.scrapeFile("Search results");
                }
                index++;
        }
}
else
{
        session.logError("===================================");
        session.logError("Cannot find input file: " + fileName);
        session.logError("===================================");
        session.stopScraping();
}

Does that get you started?

 jason on 06/13/2011 at 7:10 am