DataSets and DataRecords - HOW TO
Ok I've worked through all of the tutorials and im at a loss as to how to reference a DataSet of DataRecords from a remoteSession.
I have similar set up to the tutorial 3.
I have one script that calls another script for each pattern application.
I'm using this to extract a product list
The second scrape-file forms the URL using a session variable to identify the product id. Further the second scrape file has a extractor of named
~@DATARECORD@~ then the subextractors extract from that result.
The difference is that I am trying to communicate with DATARECORD from a Remote Java Program.
My question is this, can i reference the ~#DATARECORD#~ remotely implcitly.
i.e. is the dataset build after each proxy call to my 'Detail' scrapefile after each pattern application from my 'List' scrapefile.
I understand that other scrapists have been using the getVariable(dataSetName) call and casting to a dataset. Where do they get the dataSetName from? Would it be "DATARECORD" in my case?
or
Do I need to write a script to achieve this? Examples , how tos anyone?
Also I noticed another scraper had put a Thread.sleep(5000) command in, are there a timing constraints on retrieving data remotely?
DataSets and DataRecords - HOW TO
Not a problem. Feel free to drop us a line if we can lend a hand with anything else.
Kind regards,
Todd
DataSets and DataRecords - HOW TO
my mistake silly error i had the sequences wrong on my extractors
DataSets and DataRecords - HOW TO
That works Great, but now the product ID session variable is being lost value is being lost.
looking in the logs I can see
preliminaryUrl
http://www.mywebsite.jsp?productId=~#pid#~
but then
Resolved url
http://www.mywebsite.jsp?productId=
So whats going on?
I have a call for after the first scrape file to another scrapefile using the after event, it looks like this
session.scrapeFile("getDetails");
and I've made sure that the ~@pid@~ is a session variable, it works fine when i run it through the desktop , but when i try to run it remotely it drops the pid!
DataSets and DataRecords - HOW TO
Hi,
It sounds like you're wanting to do something similar to what we show in our fourth tutorial: here. You may want to look through that to see if it helps.
To answer your specific questions, though:
- To reference a data set via a session variable (e.g., using session.getVariable in your Java code), you would check the box labeled "Automatically save the data set generated by this extractor pattern in a session variable" under the "Advanced" tab for your extractor pattern. This would then create a session variable using the name of the extractor pattern, then retain the scraped data in that session variable as a DataSet object. That is, if you named your extractor pattern "PRODUCTS", you would then reference the data set from your Java code like this: session.getVariable( "PRODUCTS" ). Note that I'm referring to the name of the extractor pattern (labeled by the "Identifier" field) and not the extractor pattern token name (e.g., ~@DATARECORD@~).
- On the Thread.sleep question, I'm not sure which example you're looking at, but there are no timing issues that should necessitate a call to Thread.sleep.
Kind regards,
Todd Wilson