Script Failure
Newbie to your product and Java so bare with me.
I took the existing Interpreted Java script and edited it to suit a test application like so:
try
{
session.log( "Writing data to a file." );
// Open up the file to be appended to.
out = new FileWriter( "price_updates.txt", true );
// Write out the data to the file.
if (dataRecord.get( "CODE" ) != null) {
out.write( dataRecord.get( "CODE" ) + "\t" );
}
if (dataRecord.get( "HARDINESS" ) != null) {
out.write( dataRecord.get( "HARDINESS" ) + "\t" );
}
if (dataRecord.get( "DESCRIPTION" ) != null) {
out.write( dataRecord.get( "DESCRIPTION" ) + "\t" );
}
if (dataRecord.get( "COPY" ) != null) {
out.write( dataRecord.get( "COPY" ) + "\t" );
}
if (dataRecord.get( "PRODUCT" ) != null) {
out.write( dataRecord.get( "PRODUCT" ) + "\t" );
}
if (dataRecord.get( "PRICE" ) != null) {
out.write( dataRecord.get( "PRICE" ));
}
out.write( "\n" );
// Close up the file.
out.close();
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}
I set this script to be run after each scraped file has had the extraction pattern applied (1 pattern each). Field names match both in case and syntax. But I keep getting the following error:
[quote]; > : Attempt to invoke method: get() on undefined variable or class name: dataRecord : at Line: 2 : in file: If you need the rest of the error (which looks like a repeat of the script) let me know.
Script Failure
Hi,
You're running the script after the scraping session has ended, rather than after your extractor pattern has been applied (i.e., "After pattern is applied"--see my previous posting). To do this you'll add a script instance in the "Scripts" section underneath your extractor pattern. Select your script in the "Script Name" column, and "After pattern is applied" in the "When to Run" column.
Kind regards,
Todd
Script Failure
Todd - thanks for the help. . .
Some progress:
Starting scraper.
Running scraping session: 911
Processing scripts before scraping session begins.
Scraping file: "File from Hello World"
File from Hello World: Processing scripts before a file is scraped.
File from Hello World: Preliminary URL: http://motors.listings.ebay.com/_Cars-Trucks_Porsche-911_W0QQa39Z1970QQa41Z1987QQalistZa39Q2ca41QQcurcatZtrueQQfromZR42QQgcsZ13QQpfidZ2473QQsacatZ6001QQsatitleZQQsocmdZListingItemListQQsofocusZpf
File from Hello World: Resolved URL: http://motors.listings.ebay.com/_Cars-Trucks_Porsche-911_W0QQa39Z1970QQa41Z1987QQalistZa39Q2ca41QQcurcatZtrueQQfromZR42QQgcsZ13QQpfidZ2473QQsacatZ6001QQsatitleZQQsocmdZListingItemListQQsofocusZpf
File from Hello World: Sending request.
File from Hello World: Processing scripts before all pattern applications.
File from Hello World: Extracting data for pattern "Ebay_911_Output"
Processing scripts after scraping session has ended.
Scraping session finished.
Though, the output file, "products.csv" is empty. Any ideas?
Agina thanks for your patience.
Script Failure
Hi,
You need to invoke your script when the dataSet object is in scope. I'd recommend doing it after you apply your "Ebay_911_Output" extractor pattern (i.e., "After pattern is applied"). You can find more detail on variable scope in the corresponding section on this page: [url]http://www.screen-scraper.com/support/docs/using_scripts.php[/url].
Kind regards,
Todd
Script Failure
Todd - thanks for the response.
Rushed home and got everything set up - and - no dice - using this code - threw the same error. Perhaps there's some basic thing that i'm missing?
attached is the log file for review. - again thanks for your help!
Starting scraper.
Running scraping session: 911
Processing scripts before scraping session begins.
Scraping file: "File from Hello World"
File from Hello World: Processing scripts before a file is scraped.
File from Hello World: Preliminary URL: http://motors.listings.ebay.com/_Cars-Trucks_Porsche-911_W0QQa39Z1970QQa41Z1987QQalistZa39Q2ca41QQcurcatZtrueQQfromZR42QQgcsZ13QQpfidZ2473QQsacatZ6001QQsatitleZQQsocmdZListingItemListQQsofocusZpf
File from Hello World: Resolved URL: http://motors.listings.ebay.com/_Cars-Trucks_Porsche-911_W0QQa39Z1970QQa41Z1987QQalistZa39Q2ca41QQcurcatZtrueQQfromZR42QQgcsZ13QQpfidZ2473QQsacatZ6001QQsatitleZQQsocmdZListingItemListQQsofocusZpf
File from Hello World: Sending request.
File from Hello World: Processing scripts before all pattern applications.
File from Hello World: Extracting data for pattern "Ebay_911_Output"
Processing scripts after scraping session has ended.
Processing script: "NewerScript"
911: An error occurred while processing the script: NewerScript
911: The error message was: The application script threw an exception: java.lang.NullPointerException BSF info: null at line: 0 column: columnNo
Scraping session finished.
Current Output Code
// To be placed at the end of a scraping session
// An array of the names of elements of your datarecords. See tutorial3.
String[] keys = { "ITEMNUMBER", "ITEMLINK", "ITEMIMAGE", "MFG", "MODEL", "MODELYEAR", "MILEAGE", "BIDS", "PRICE", "LISTINGDATE" };
// Function to avoid any null problems, if one value was not scraped
// in a record.
String checkNull(String s) {
if (s == null)
return "";
return s;
}
// Get the dataset. Make sure that the extractor patterns were saved
// to a data set.
com.screenscraper.common.DataSet dataset = session.getVariable("ITEMNUMBER");
// write to a file called 'products.csv'
FileWriter out = new FileWriter("products.csv");
// Loop through all data records in the data set
for (int i = 0; i < dataset.getNumDataRecords(); i ) {
tmpDataRecord = dataSet.getDataRecord(i);
// Loop through all elements of a datarecord as specified in keys[].
for (int j = 0; j < keys.length; j ) {
out.write(checkNull(tmpDataRecord.get(keys[j])));
out.write("\t");
}
out.write("\n");
}
// Close the file writer and you are done.
out.close();
Script Failure
Hi,
I think you're really close on this one. Looks like you just need to pull the data record from the data set before referencing its values.
Instead of this:
for (int i = 0; i < dataset.getNumDataRecords(); i ) {
// Loop through all elements of a datarecord as specified in keys[].
for (int j = 0; j < keys.length; j ) {
out.write(checkNull(dataset.get(i, keys[j])));
out.write("\t");
}
out.write("\n");
}
Try this:
for (int i = 0; i < dataset.getNumDataRecords(); i ) {
tmpDataRecord = dataSet.getDataRecord(i);
// Loop through all elements of a datarecord as specified in keys[].
for (int j = 0; j < keys.length; j ) {
out.write(checkNull(tmpDataRecord.get(keys[j])));
out.write("\t");
}
out.write("\n");
}
Feel free to write back if that doesn't seem to do the trick.
Kind regards,
Todd Wilson
Script Failure
Hi,
Newbie, here. 1st off - this is a fantastic product - i've gotten everything working except the save-off functionallity. I've tried two methods of saving data posted in the forums and have come to my wits end.
I tried this method, based on this post: http://www.screen-scraper.com/forum/phpBB2/viewtopic.php?t=74&highlight=undefined+variable++class+name&sid=a1582b08d76ef612a3b7dcb55589f0b2
. . .which gives me this error:
The error message was: The application script threw an exception: java.lang.NullPointerException BSF info: null at line: 0 column: columnNo
And my code. . .scraping an ebay motors page is below. . .
(Note - all tokens are saved as session variables)
// To be placed at the end of a scraping session
// An array of the names of elements of your datarecords. See tutorial3.
String[] keys = { "ITEMNUMBER", "ITEMLINK", "ITEMIMAGE", "MFG", "MODEL", "MODELYEAR", "MILEAGE", "BIDS", "PRICE", "LISTINGDATE" };
// Function to avoid any null problems, if one value was not scraped
// in a record.
String checkNull(String s) {
if (s == null)
return "";
return s;
}
// Get the dataset. Make sure that the extractor patterns were saved
// to a data set.
com.screenscraper.common.DataSet dataset = session.getVariable("ITEMNUMBER");
// write to a file called 'products.csv'
FileWriter out = new FileWriter("products.csv");
// Loop through all data records in the data set
for (int i = 0; i < dataset.getNumDataRecords(); i++) {
// Loop through all elements of a datarecord as specified in keys[].
for (int j = 0; j < keys.length; j++) {
out.write(checkNull(dataset.get(i, keys[j])));
out.write("\t");
}
out.write("\n");
}
// Close the file writer and you are done.
out.close();
Any help is appreciated.
Best regards.
Script Failure
Got it thanks.
G.
Script Failure
That's good to know but I was after the User Agent information the scraper leaves when it scrapes a page off the target website.
Script Failure
Understood. So I determined this code:
session.getVariable("PRODUCTS");
is used to look at a particular session (created with each file to be scraped). I used the same name for each extraction pattern and then ran the loop at the end of the scraping session for each file. It works pretty slick. Now I have to write the glue app to plug the output into my db. Fortunately, PHP is something I know well. ;o)
One last question. What footprints (UI signature) does the application leave and can I change this?
Thanks for you help.
Script Failure
gbanse,
> 1. why was the loop necessary? I had set the script to run after each extraction pattern was run. Wouldn't this effectively work the same?
It is a matter of preference. You can append to the file after every pattern match, or you can write out the whole dataset after all files have been scraped. If you do write to the file after every match, then it would be better to use the object "dataRecord", which is in scope after each match.
> 2. In my setup I have 26 scraped files, each with it's own extraction pattern. Why only one call to check a scraped file session was saved as a session variable?
If each page is a different type of data like products and addresses then the example script would not be what you would want. If all pages have the same type of data and all append to the same dataset, then all the data would be available from that one variable. Again, it really depends on the pages you scrape and the logic needed.
Brent
[email protected]
Script Failure
With a bit of modifying I made the script you posted work. Fantastic - thank you.
A few follow up questions:
1. why was the loop necessary? I had set the script to run after each extraction pattern was run. Wouldn't this effectively work the same?
2. In my setup I have 26 scraped files, each with it's own extraction pattern. Why only one call to check a scraped file session was saved as a session variable?
com.screenscraper.common.DataSet dataset = session.getVariable("PRODUCTS");
or should there be one of these for each scraped file?
Thanks again.
Script Failure
gbanse,
I am glad to see that you were able to figure out your first troubles with scripting in Java using screen-scraper.
In my experience with scripting in Interpreted Java, I have found that when the message is 'null', then this means that there is a NullPointerException. In other words some value is null that should not be.
In this case it could be that the values returned by some of you session.getVariable calls resulted in a null value. I would check to see if you are properly saving some of those values (edit token to save the value to a session variable).
If you were trying to print out a dataset (several items scraped from one or several pages), then here is the way I like to do that as a an example for all who read this forum (caution it has not been tested):
// An array of the names of elements of your datarecords. See tutorial3.
String[] keys = {"NAME", "PRICE", "DESCRIPTION"};
// Function to avoid any null problems, if one value was not scraped
// in a record.
String checkNull(String s) {
if (s == null)
return "";
return s;
}
// Get the dataset. Make sure that the extractor patterns were saved
// to a data set.
com.screenscraper.common.DataSet dataset = session.getVariable("PRODUCTS");
// write to a file called 'products.csv'
FileWriter out = new FileWriter("products.csv");
// Loop through all data records in the data set
for (int i = 0; i < dataset.getNumDataRecords(); i++) {
// Loop through all elements of a datarecord as specified in keys[].
for (int j = 0; j < keys.length; j++) {
out.write(checkNull(dataset.get(i, keys[j])));
out.write("\t");
}
out.write("\n");
}
// Close the file writer and you are done.
out.close();
Brent
[email protected]
Script Failure
Ok, I realized some mistakes in the first post. Here's the latest attempt:
try
{
session.log( "Writing data to a file." );
// Open up the file to be appended to.
out = new FileWriter( "price_updates.txt", true );
// Write out the data to the file.
out.write( session.getVariable( "CODE" ) + "\t" );
out.write( session.getVariable( "HARDINESS" ) + "\t" );
out.write( session.getVariable( "DESCRIPTION" ) + "\t" );
out.write( session.getVariable( "COPY" ) + "\t" );
out.write( session.getVariable( "PRODUCT" ) + "\t" );
out.write( session.getVariable( "PRICE" ) );
out.write( "\n" );
}
catch( Exception e )
{
session.log( "An error occurred while writing the data to a file: " + e.getMessage() );
}
Now I'm down to one last error:
An error occurred while writing the data to a file: null<code>