Help with output
I have ran into a problem with the way I am currently outputting my data. I am currently writing the data to csv and after each page is scraped it saves it as the page name as a separate csv. However I want to make this one big file instead of multiple separate files. For example I have multiple files named a.csv, b.csv, c.csv, ... In each file it contains a schedule which includes multiple dates, locations, and times. There is only one instance on each page of the page title (a, b, c, ...). The problem I run into is when I try to include that into the csv it will only output one full record. So I need to figure out a way to have the page title written to csv for the number of instances there are of the schedule data.
My current file looks like the following: "a.csv"
(DATE,LOCATION,TIME)
Date1,Location1,Time1
Date2,Location2,Time2
Date3,Location3,Time3
I want it to look like this: "allschedules.csv"
(PAGE_TITLE,DATE,LOCATION,TIME)
A,Date1,Location1,Time1
A,Date2,Location2,Time2
A,Date3,Location3,Time3
B,Date1,Location1,Time1
B,Date2,Location2,Time2
Is there any way that I can do this my code I am using is below.
--------------------------------------------------------------------
outputFile = session.getVariable ("PAGE_TITLE"); + ".csv"
// Error catching.
try
{
File file = new File( outputFile );
fileExists = file.exists();
// Open up the file to be appended to.
out = new FileWriter( outputFile, true );
session.log( "Writing data to a file." );
//this piece of code is responsible to write out the headers only 1 time.
if (!fileExists)
{
// Write out the headers.
out.write("\"" + "DATE" + "\"" + ",");
out.write("\"" + "LOCATION" + "\"" + ",");
out.write("\"" + "TIME" + "\"" + ",");
out.write( "\n" );
}
// Write columns.
out.write( session.getVariable( "DATE" )+ "," );
out.write( session.getVariable( "LOCATION" )+ "," );
out.write( session.getVariable( "TIME" ) + "," );
// Close up the file.
out.close();
// Clear variables. you only need to clear session variables because dataRecord variables don't persist
session.setVariable("DATE","");
session.setVariable("LOCATION","");
session.setVariable("TIME","");
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}
Hey there-- I'm going to copy
Hey there-- I'm going to copy and paste your script into my reply, and I'll highlight the parts that I would add in green, and the parts that I would change in red:
outputFile = session.getVariable ("OUTPUT_FILE_NAME") + ".csv"
// Error catching.
try
{
File file = new File( outputFile );
fileExists = file.exists();
// Open up the file to be appended to.
out = new FileWriter( outputFile, true );
session.log( "Writing data to a file." );
//this piece of code is responsible to write out the headers only 1 time.
if (!fileExists)
{
// Write out the headers.
out.write("PAGE_TITLE,");
out.write("DATE,");
out.write("LOCATION,");
out.write("TIME,");
out.write("\n");
}
// Write columns.
out.write( "\"" + session.getVariable( "PAGE_TITLE" ) + "\"," );
out.write( "\"" + session.getVariable( "DATE" ) + "\",");
out.write( "\"" + session.getVariable( "LOCATION" ) + "\",");
out.write( "\"" + session.getVariable( "TIME" ) + "\",");
out.write( "\n" );
// Close up the file.
out.close();
// Clear variables. you only need to clear session variables because dataRecord variables don't persist
session.setVariable("DATE","");
session.setVariable("LOCATION","");
session.setVariable("TIME","");
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}
----------------------------------------------
That should do it. I added the extra "\"" parts for writing the columns because if you were to have any commas in your scraped data, that would mess up the columns. Putting the quotes around it will stop that.
Notice that I'm *not* clearing the "PAGE_TITLE" variable. If you clear the PAGE_TITLE variable, then it won't be available the second and third and fourth times that you call this script on the same page. Also, you'll need to have a new variable called "OUTPUT_FILE_NAME" (or whatever you want to call it). Set it in some initializing script so that it's ready and available during the whole scrape. It will be the name of the single file that you want to write out to.
If you need the single file to have a unique name from day to day, you can try doing something like this to set the "OUTPUT_FILE_NAME" variable:
session.setVariable("OUTPUT_FILE_NAME", "allschedules - " + (new Date()).toString());
This would set the name of your output file to "scrape results" plus the current date. Only do this part in the initialize script; you only want to set the name of the output file just once. If you were to make the mistake of calling this line at the top of your write-to-file script, then it will come up with a new name every time you call it, since the hours, minutes, and seconds are also included in the date that this would append to the file name.
Is that what you're looking for? Let me know if you need more assistance :)
Tim