Home » Forums » screen-scraper » screen-scraper public support

Adding to ecommerce tutorial

Hey guys, I'm updating the ecommerce tutorial to include 2 text files. One for letters a, b, and c, and the other that contains the 1000 most common last names. I'm trying to search the first name and last name fields of a search form. However, what is happening is it takes the first last name (aaron for example) and loops through the file a, b, and c, correctly.

So, it does 3 searches correctly. However, when it's time to switch to the second last name (abrams for example), it doesn't read that second name or any other name for that matter, it just says search done. Wondering what I'm missing. Any help would be appreciated. Hopefully I'm just missing something silly. I can also include the .sss file if that would help.

Here's the code.

// Create a file object that will point to the file containing
// the search terms.
File inputFile = new File( "last_names.txt" );
File inputFile2 = new File( "a_z.txt" );

// These two objects are needed to read the file.
FileReader in = new FileReader( inputFile );
BufferedReader buffRead = new BufferedReader( in );

FileReader in2 = new FileReader( inputFile2 );
BufferedReader buffRead2 = new BufferedReader( in2 );

// Read the file in line-by-line.  Each line in the text file
// will contain a search term.
while( ( searchTerm = buffRead.readLine() )!=null)
{

        while( ( searchTerm2 = buffRead2.readLine() )!=null)
{
        session.log("term 1 = " + searchTerm);
  session.log("term 2 = " + searchTerm2);
    // Set a session variable corresponding to the search term.
   session.setVariable( "SEARCH", searchTerm );
    session.setVariable( "SEARCH2", searchTerm2 );
    // Remember we need to initialize the PAGE session variable, just
    // in case we need to iterate through multiple pages of search results.
    // We begin at page 1 for each search.
    session.setVariable( "PAGE", "1" );

    // Get search results for this particular search term.
    session.scrapeFile( "Search results" );
}

}

// Close up the objects to indicate we're done reading the file.
in.close();
buffRead.close();
in2.close();
buffRead2.close();

And here is the log.

Starting scraper.
Running scraping session: LVSU
Processing scripts before scraping session begins.
Skipping the following script because it's disabled: "Shopping Site--initialize session"
Scraping file: "Home"
Home: Processing scripts before a file is scraped.
Processing script: "Read search terms"
term 1 = caldwell
term 2 = a*
Scraping file: "Search results"
Search results: Processing scripts before a file is scraped.
Search results: Preliminary URL: https://directory.LVSU.edu/findpeople.php
Search results: Using strict mode.
Search results: POST data: ChooseMatch=student&firstname=a*&lastname=caldwell&name_n=&key=value
Search results: Resolved URL: https://directory.LVSU.edu/findpeople.php
Search results: Sending request.
Search results: Processing scripts before all pattern applications.
Search results: Applying extractor pattern: email
Search results: Extracting data for pattern "email"
Search results: The following data elements were found:
email--DataRecord 0:  
THE_STONES=caldwell.474
Search results: Processing scripts after a pattern application.
Processing script: "Write data to a file"
Writing data to a file.

Search results: Processing scripts after a pattern application.
Processing script: "Write data to a file"
Writing data to a file.
Search results: Processing scripts once if pattern matches.
Search results: Processing scripts after all pattern applications.
Search results: Processing scripts after a file is scraped.

term 1 = caldwell
term 2 = b*
Scraping file: "Search results"
Search results: Processing scripts before a file is scraped.
Search results: Preliminary URL: https://directory.LVSU.edu/findpeople.php
Search results: Using strict mode.
Search results: POST data: ChooseMatch=student&firstname=b*&lastname=caldwell&name_n=&key=value
Search results: Resolved URL: https://directory.LVSU.edu/findpeople.php
Search results: Sending request.
Search results: Processing scripts before all pattern applications.
Search results: Applying extractor pattern: email
Search results: Extracting data for pattern "email"
Search results: The following data elements were found:
email--DataRecord 0:  
THE_STONES=caldwell.470
Search results: Processing scripts after a pattern application.
Processing script: "Write data to a file"
Writing data to a file.

Search results: Processing scripts after a pattern application.
Processing script: "Write data to a file"
Writing data to a file.
Search results: Processing scripts once if pattern matches.
Search results: Processing scripts after all pattern applications.
Search results: Processing scripts after a file is scraped.
 
term 1 = caldwell
term 2 = c*
Scraping file: "Search results"
Search results: Processing scripts before a file is scraped.
Search results: Preliminary URL: https://directory.LVSU.edu/findpeople.php
Search results: Using strict mode.
Search results: POST data: ChooseMatch=student&firstname=c*&lastname=caldwell&name_n=&key=value
Search results: Resolved URL: https://directory.LVSU.edu/findpeople.php
Search results: Sending request.
Search results: Processing scripts before all pattern applications.
Search results: Applying extractor pattern: email
Search results: Extracting data for pattern "email"
Search results: The following data elements were found:
email--DataRecord 0:  
THE_STONES=caldwell.567
Search results: Processing scripts after a pattern application.
Processing script: "Write data to a file"
Writing data to a file.


Search results: Processing scripts after a pattern application.
Processing script: "Write data to a file"
Writing data to a file.
Search results: Processing scripts once if pattern matches.
Search results: Processing scripts after all pattern applications.
Search results: Processing scripts after a file is scraped.
Skipping the following script because it's disabled: "Read search terms"
Home: Preliminary URL: https://directory.lvsu.edu/findpeople.php
Home: Using strict mode.
Home: Resolved URL: https://directory.lvsu.edu/findpeople.php
Home: Sending request.
Home: Processing scripts after a file is scraped.
Processing scripts after scraping session has ended.
Scraping session "LVSU" finished.

Thanks again for any insight!

‹ An error occurred while writing the data to a file: null filter duplicates - cannot get it to work! ›
ilovescrape on 03/06/2011 at 10:01 pm

I think your nesting might be off...

I actually stumbled across your post when I was looking to do something similar. I think your problem might have something to do with the position of the in2, buffread2 and inputfile2 lines. These need to be nested within the first while loop, so:

// Create a file object that will point to the file containing
// the search terms.
File inputFile = new File( "last_names.txt" );

// These two objects are needed to read the file.
FileReader in = new FileReader( inputFile );
BufferedReader buffRead = new BufferedReader( in );

// Read the file in line-by-line.  Each line in the text file
// will contain a search term.
while( ( searchTerm = buffRead.readLine() )!=null)
{
//JS: nesting begins here...
 File inputFile2 = new File( "a_z.txt" );
 FileReader in2 = new FileReader( inputFile2 );
 BufferedReader buffRead2 = new BufferedReader( in2 );
 while( ( searchTerm2 = buffRead2.readLine() )!=null)
{
...

This seemed to work for me...hope this helps you!

Regards,
Justin

 Justin_S on 09/13/2011 at 2:10 pm

Have you tested to see if

Have you tested to see if search Term is null after the nested loop? With a session.log. And maybe just test to see if the other loop is working by itself and outputting the different values by itself just to see where the error is coming from. i.e commented out the inner loop.

 seamus1982 on 03/07/2011 at 1:01 pm