Free code: How to tell what was passed to a scrape.
Below is included some code I call from my "__globaldefault" script before the scraping session starts -- it builds a table containing the scrape name and variables passed. Saves it all in a table called TABLE0. Hope this helps some folks as much as it did me.
Disclaimer: session.getScrapingSessionState().getVariables().keys() is considered experimental/prototype code, so if it doesn't work, don't blame me. Or screen-scraper. :)
Oh and did I mention I'm a noob to Java? So if the code isn't textbook quality, it's not on purpose!
myDataSet = new DataSet();
int i;
String s;
Enumeration enum;
// Set some generic output
// myDataRecord.put( "INFO" , "SCRAPENAME=" + session.getName());
session.setVariable("_server.sessioname",session.getName());
enum = session.getScrapingSessionState().getVariables().keys();
while (enum.hasMoreElements()){
e = enum.nextElement();
if (e.getClass().getName().equals("java.lang.String")) {
s = (String) e;
if( session.getVariable( s ) != null ){
if (session.getVariable(s).getClass().getName().equals("java.lang.String")) {
String thisvalue = session.getVariable( s );
if (thisvalue.length() != 0) {
myDataRecord = new DataRecord();
myDataRecord.put( "PARAMETERS" , s + "=" + thisvalue);
myDataSet.addDataRecord( myDataRecord );
session.log(s + "=" + thisvalue );
}
}
}
}
}
// Save it all in Table0
session.setVariable( "TABLE0", myDataSet );
Depricated?
I'm guessing the "experimental"
session.getScrapingSessionState().getVariables().keys()
Has been depricated. If so, is there an equivalent in the current API? I need a way to find out all of the parameters passed to a scrape.
Nevermind, it appears(?) to still be there. But is there a more proper way to go about this?
Free code: How to tell what was passed to a scrape.
Now that I have your attention, my question:
I want to check the passed variables, and if the right ones weren't passed, I want to end the session. session.stopScraping() does that nicely, but I guess what I *really* want is to skip the rest of the session and return what I've gotten so far.
Yes, I know I can use scripts to call the scrapes/patterns/etc and wrap ifs around them, but I'd really like a silver bullet in one script I can just call once per scrape. I'm lazy that way. Thoughts?
edited:
Ok, I'm back and it appears I ran my script in the wrong place. I'll share that one eventually if people are interested. session.stopScraping(), when called from a script run before the whole scrape, not just a scrapeable page -- returns table0 nicely.
Robert, Sorry for not being
Robert,
Sorry for not being more help along the way. It sounds like you have your problem solved. Please post any code you would like to share and let us know if you have any other questions.
-Scott