Ignoring HTML in scrapping session results

Hi Jason,

Okay, not sure what I did but I fixed it, and it now works on my localhost.

I gave up trying externally online.

I'm not going to try to work it out as it's been many long evening trying to get it to work, and it does now ...

So 1000 thanks for making my life lots quicker with this file.

Two questions though.

I have changed the write to file to include the this to write the localhost url:

out.write("\n");
}

// Write columns
for (i=0; i {
var = names[i];
var = var.toUpperCase();
var = var.replaceAll("\\s", "_");
out.write(fixString(dataRecord.get(var)));
if (i out.write("\t");

}
out.write(scrapeableFile.getCurrentURL() + "\t" );
out.write( "END");
out.write( "\n" );

// Close up the file
out.close();
}

The thing is it writes it twice for some reason? Not sure if I put it in the right position.

Second,

If there are urls in the list that I import to scrape that are wrong, in the way I used to do it, it would have written START END and I think it was NULL in the missing fields in the line so When I compared the list of import urls with the final list it would be the same length and match.

Now if there is a missing/wrong url it ignores it and goes to write the next line, which will mean sorting out after the scrape is done.

How do I get it to write something in each line?

Here is the old csv writing script that did this

I am using this to write to csv, is it not working because of this, it runs after each pattern match.

FileWriter out = null;

try
{
session.log( "Writing data to a file." );

// Open up the file to be appended to.
out = new FileWriter( "data-from-files-in-local-folder.txt", true );

// Write out the data to the file.

out.write( "START" + "\t" );
out.write(scrapeableFile.getCurrentURL() + "\t" );
out.write( dataRecord.get( "NAME" ) + "\t" );
out.write( dataRecord.get( "LOCALITY" ) + "\t" );
out.write( "END" + "\n" );

// Close up the file.
out.close();
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}