How to output a value of Null to .csv when "No matches were made by any of the extractor patterns ..."
Hi,
I have a lot of urls to scrape!
I have a scraping session that scrapes 70% of the urls into a .csv file, no problem with the scraping session.
The problem is that when the session reaches a url that it reports back "No matches were made by any of the extractor patterns associated with this scrapeable file."
it moves onto the next url and does not write Null or blank ... as its output into the .csv
WHAT I WANT:
I want my "write to .csv script" to write a value when no matches were made by any of the extractor patterns associated with this scrapeable file.
That way I can clearly see which urls were not scraped.
Even better would be if the "write to .csv output script could write" the name of the url.
Here's a record of the scraping session:
Starting scraper.
Running scraping session: scraping-files-in-local-folder
Processing scripts before scraping session begins.
Processing script: "importing-urls-from-txt-file"
Scraping file: "scraping-files-in-local-folder"
Scraping local file: /Users/me/Downloads/info/shopping-14.html
scraping-files-in-local-folder: Processing scripts before all pattern applications.
scraping-files-in-local-folder: Extracting data for pattern "scraping pattern"
scraping-files-in-local-folder: The pattern did not find any matches.
scraping-files-in-local-folder: scraping pattern: Processing scripts once if no matches.
scraping-files-in-local-folder: scraping pattern: Processing scripts after all pattern applications.
scraping-files-in-local-folder: Warning! No matches were made by any of the extractor patterns associated with this scrapeable file.
Processing scripts after scraping session has ended.
Processing scripts always to be run at the end.
Scraping session "scraping-files-in-local-folder" finished.
didn't work
It didn't write out a default value.
My script is this and I put your code at the top of it. Any suggestions. Thanks:
FileWriter out = null;
try
{
session.log( "Writing data to a file." );
// Open up the file to be appended to.
out = new FileWriter( "data-from-files-in-local-folder.txt", true );
// Write out the data to the file.
out.write( dataRecord.get( "H1" ) + "\n" );
// Close up the file.
out.close();
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}
thanks
Thanks.
The script runs after the pattern but the code looks like it will do the job.
I'll let you know tomorrow if I need more help/if it worked.
Thanks again!!!
What you need is make your
What you need is make your write to file script run "after the pattern is applied" and
if (dataSet.getNumDataRecords()>0)
{
for (i=0; i<dataSet.getNumDataRecords(); i++)
{
dr = dataSet.getDataRecord(i);
//Write out items from dr
}
}
else
{
// Write out default
}