Home » Tips, Tricks, & Samples » Samples » Script Repository » Iteration

Scrape Only Recent Information

This script is designed to check how recent a post or advertisement is. If you were gathering time sensitive information and only wanted to reach back a few days then this script would be handy. After evaluating the date there will be a section for calling other scripts from inside this script.

//start with these imports
import java.util.Date;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.lang.*;
import java.util.*;
import java.io.*;

// Function to parse the passed string into a date
makeDate(date)
{
//This is the format for your date. It is in the April 20, 1999 format
formatter = new SimpleDateFormat("MMM d, yyyy");

//some other options instead of blank could be null, N/A, etc. Really it just depends on how the site is structured.
    if (date.equals("BLANK")){
        session.log(" ---NO ATTEMPT TO PARSE BLANK DATE");
    }
//if it is not blank go ahead and parse the data and apply the Format above. This will also print the date to the log.
    else{
        date = (Date)formatter.parse(date);
        session.log(" +++Parsed date " + date);
    }
    return date;
}

// Function to get current date
oldestDate(){
    // Set number of days to minus from current date.
    minusDays = -5;

    // Get the current date or instance, then you are going to add a negative amount of days. If that seems strange
     // Just trust us. This is not a double negative thing.
    Calendar rightNow = Calendar.getInstance();
    rightNow.add( Calendar.DATE, minusDays );

    // Substitute the Date variable endDate for rightNow becuase it makes more sense to
     // Return endDate than a variable named rightNow which is 5 days in the past.
    Date endDate = rightNow.getTime();
    session.log("The end date is: " + endDate);
    return endDate;
}

// Parse posted date. you are getting this posted date from a dataRecord.
// if you were getting it from a session variable it would say session.getVariable("POSTED_DATE")
posted = makeDate(dataRecord.get("POSTED_DATE"));

// Parse the current Date and return it in a format that you can compare to the advertisement or post date.
desired = oldestDate();

// Compare the two.<br />
 if (posted.after(desired) || posted.equals(desired))
{
    session.log ("AD IS FRESH. SCRAPING DETAILS.");

    // If you are keeping track of URLs this will get it from the scrapeable file.
    session.setVariable ("SOURCE_URL", scrapeableFile.getCurrentURL() );

    // This is the place in the code where you would execute additional scripts.
    session.executeScript("Your script name here");
    session.executeScript("Your second script name here");
}
else{
    session.log("Posted is too old");
}

Hopefully it is evident that the above code is useful in comparing todays date against a previous one. Depending on your needs you might consider developing a script which will move your scraping session on after it reaches a certain date in a listing. For example if you were scraping an auction website for many terms you might want to move on to the next term after you have reached a specified date for the listings. What are some other ways this script could be useful?

scraper on 07/16/2010 at 4:25 pm