The most easily maintainable way of logging in is to have multiple scrapeable files which do the logging in then go to the page to be scraped. However, there already exists a method for setting one's headers. Please see Using Scripts in our documentation. One can set the headers on the scrapeableFile object. This is a method that has been in screen-scraper for a while and hopefully you find useful. Be sure to understand the scope of this object for when to call a script that would set the cookie information.
I managed to invoke a script (see below) that calls files in a particular order (ampersand seperated filenames)...
import java.util.StringTokenizer;
//Scrape the login file first
session.setVariable("USERNAME",URLEncoder.encode("username"));
session.setVariable("PASSWORD",URLEncoder.encode("password"));
session.scrapeFile("Login");
//Get the Files to Scrape from the Session
scrapeFileNames=URLDecoder.decode(session.getVariable("SCRAPEFILES"));
// Tell the scraping session to scrape the required files.
StringTokenizer fileToken = new StringTokenizer(scrapeFileNames, "&");
One cannot externally call a specific scrapeableFile through Java. However, one can pass in to a scraping session strings containing any thing, like URL's or order of scrapableFiles. The logic of the scrape was meant to be contained withing scripts within a scraping session.
Also, from within scripts one can call methods written in Java. To do so create a jar and save it in the directory lib/ext within the screen-scraper directory. One can then import any class and call the methods directly from scripts. This is especially helpful for accessing databases.
Web Data Extraction
Invoke a Scrapeable file from Java
David,
The most easily maintainable way of logging in is to have multiple scrapeable files which do the logging in then go to the page to be scraped. However, there already exists a method for setting one's headers. Please see Using Scripts in our documentation. One can set the headers on the scrapeableFile object. This is a method that has been in screen-scraper for a while and hopefully you find useful. Be sure to understand the scope of this object for when to call a script that would set the cookie information.
Good luck,
Brent
[email protected]
Invoke a Scrapeable file from Java
Thanks Brent,
I managed to invoke a script (see below) that calls files in a particular order (ampersand seperated filenames)...
import java.util.StringTokenizer;
//Scrape the login file first
session.setVariable("USERNAME",URLEncoder.encode("username"));
session.setVariable("PASSWORD",URLEncoder.encode("password"));
session.scrapeFile("Login");
//Get the Files to Scrape from the Session
scrapeFileNames=URLDecoder.decode(session.getVariable("SCRAPEFILES"));
// Tell the scraping session to scrape the required files.
StringTokenizer fileToken = new StringTokenizer(scrapeFileNames, "&");
while(fileToken.hasMoreTokens()){
session.scrapeFile(fileToken.nextToken());
}
I need to set some cookies on the session, so every time I scrape I need to call a login page.
I see from the other posts that cookies are a bit of an issue; are there plans to enable setting of header information (e.g. Cookie) ?
Regards,
David
Invoke a Scrapeable file from Java
dtrigwell,
One cannot externally call a specific scrapeableFile through Java. However, one can pass in to a scraping session strings containing any thing, like URL's or order of scrapableFiles. The logic of the scrape was meant to be contained withing scripts within a scraping session.
Also, from within scripts one can call methods written in Java. To do so create a jar and save it in the directory lib/ext within the screen-scraper directory. One can then import any class and call the methods directly from scripts. This is especially helpful for accessing databases.
Brent Wenerstrom
[email protected]