screen-scraper public support
Language issues
I was wondering what a good resource would be for learning this Interpreted Java. It just seems like nothing I am used to is working. I am a JavaScript developer but I can't even get it to log the length of a string. Any ideas? Thank you.
Problem with Cookies
SS doesn't seem to be setting a cookie correctly on a site I'm working on.
I hit the front page of the site, and this is the response (part of it, anyway) that I get back:
scraping a list of imported filenames (from text file)
new user, very little interpreted java code experience.
I have a scraping session set up correctly to scrape, and output my tab delimited text file.
I have set the initializing script to use variable:
runnableScrapingSession.setVariable( "FILE", "1" );
This variable is called by the scrape session URL:
C:\pages\~#FILE#~.html
This works fine. Now I need to extend it to go thru a list of filenames.
How can I load a list of names from a text file, have it scrape each, and append the scraped data to my results text file.
Couple of Questions about upgrading to Pro Edition
I've been working on a project with the free edition, and I've loved it. I'm very impressed with the quality of the product, and the amazing level of support through this forum for a program I haven't paid for!
However, I'm now getting in to some areas where the pro edition will be necessary, and have a couple of questions about it.
First, will I be able to transfer the sessions and scripts I have inthe basic edition over to the pro edition once it is installed?
Free code: How to tell what was passed to a scrape.
Below is included some code I call from my "__globaldefault" script before the scraping session starts -- it builds a table containing the scrape name and variables passed. Saves it all in a table called TABLE0. Hope this helps some folks as much as it did me.
Disclaimer: session.getScrapingSessionState().getVariables().keys() is considered experimental/prototype code, so if it doesn't work, don't blame me. Or screen-scraper. :)
Oh and did I mention I'm a noob to Java? So if the code isn't textbook quality, it's not on purpose!
Memory issues
I have a complex scraping session that loops multiple times with parameters called from a database. It begins running slowly afer a while, and then stops producing a memory error.
I know you have said in other posts that the variables are kept in memory until the scraping session is either stopped or the variables are specifically zeroed out.
How do you zero out variables? I've seen you mention it but can't find the documentation for it.
Uploading a file
Is there a way to upload a file via a web form with the type=file designation?
I have a scrape where the data that it pulls can be over 200,000 lines. As this file is too large to upload in a single csv file, and posting it to a form a line at a time is much slower, I would like to upload it automatically to a php file upload form that will insert it into the database periodically every 5,000 lines or so.
I can't sem to see how to get it to post the file to a form, though. Is this possible?
multiple tasks
can Screen-Scraper run multiple tasks and see output, logs or debug immediately. :?:
Preferred Deployment Method
Just curious to know what the best, most proper way to deploy scrapes would be.
1. Stop the server, copy the whole database up, start the server
2. Use the "import" folder (if I understand the concept correctly)
3. Post each scrape to the server using SOAP as they are updated.
Am I missing any methods? And which one would be the cleanest?
Help for a beginning scraper?
I have something I need to scrape like this: