Text File Column Headers

Hi Scott,

It may be that the answer to your question is this:


http://www.mysite/similar_folder/unique_page~#PAGE#~.html


That is, you can embed session variables anywhere in the URL, which, depending on what's stored in the session variables, could yield URL's like these:


http://www.mysite/similar_folder/unique_pageA.html
http://www.mysite/similar_folder/unique_pageBA.html


That may only be the first part of your question. The second part may deal with getting the "words" that you'd like to embed in the URL. The easiest way would be to extract them from a page. For example, you may have all of those URL's embedded in a web page. You might then apply an extractor pattern that grabs the unique portion of each URL (e.g., "A" or "BA"), and saves that unique portion in a session variable (e.g., "PAGE"). You could then embed them in your URL like so:


http://www.mysite/similar_folder/unique_page~#PAGE#~.html


If you can't extract the unique portions of the URL from the page, you could also read them in from a file or URL. We provide an example of doing this here: here.

Hopefully that answers your questions. Feel free to post back, if not.

Best wishes,

Todd

Hi Scott,

You've got some extra parentheses. Try changing your script to this:

FileWriter out = null;

try
{
session.log( "Write SSS Headers to file" );

// Open up the file to be appended to.
out = new FileWriter( "SSS.txt", false );

// Write out the data to the file.
out.write( "4pc_Place" + "\t" );
out.write( "5pc_Place" + "\t" );
out.write( "4pc_Dinner" + "\t" );
out.write( "5pc_Dinner" + "\t" );
out.write( "46pc_Place" + "\t" );
out.write( "46pc_Dinner" + "\t" );
out.write( "66pc_Place" + "\t" );
out.write( "66pc_Dinner" );
out.write( "\n" );

// Close up the file.
out.close();
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
};

Best,

Todd

Hi,

I believe I understand what you're asking, but feel free to correct me, if not. The key here is the constructor for the FileWriter object. To start a fresh file (overwriting an existing file) you would create the object like so:

out = new FileWriter( "dvds.txt", false );

To append to an existing file you would instead do the following:

out = new FileWriter( "dvds.txt", true );

Oftentimes we'll write out column headers by invoking a script that gets invoked before a scraping session begins, which might look something like this:

FileWriter out = null;

try
{
session.log( "Writing headers to a file." );

// Open up the file to be created.
out = new FileWriter( "dvds.txt", false );

// Write out the data to the file.
out.write( "TITLE" + "\t" );
out.write( "PRICE" + "\t" );
out.write( "MODEL" + "\t" );
out.write( "SHIPPING_WEIGHT" + "\t" );
out.write( "MANUFACTURED_BY" );
out.write( "\n" );

// Close up the file.
out.close();
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}

If you invoke that script before the scraping session begins it will start out your file with only the column headers. After that, you'll likely call a script more like this one:

FileWriter out = null;

try
{
session.log( "Writing data to a file." );

// Open up the file to be appended to.
out = new FileWriter( "dvds.txt", true );

// Write out the data to the file.
out.write( dataRecord.get( "TITLE" ) + "\t" );
out.write( dataRecord.get( "PRICE" ) + "\t" );
out.write( dataRecord.get( "MODEL" ) + "\t" );
out.write( dataRecord.get( "SHIPPING_WEIGHT" ) + "\t" );
out.write( dataRecord.get( "MANUFACTURED_BY" ) );
out.write( "\n" );

// Close up the file.
out.close();
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}

Which will simply append data to the file as it gets scraped. The headers will remain in-tact throughout the process.

Just let me know if I can clarify any of that.

Kind regards,

Todd Wilson