screen-scraper public support
Scrape an entire site?
Is there any way to scrape the contents every page on the site? Automatically scrape all the pages in a domain??
HTTPS, VISTA, IE7
I can't seem to get a connection at all to any HTTPS website. Is there anyone else having trouble scraping ANY HTTPS page? I am running a Vista Ultimate with IE7 and SS 4.0. I used to be able to access HTTPS sites with SS 2.7.2. SS 4.0 is just saying there is an error nothing else. I've tried connecting to a variety of site and I have the same problem.
My settings in IE7 are to point to my IP address port 8777.
Thanks,
Larry
Pausing/Timing
I've setup a file to rotate by searm terms based on a search.csv file - the file usally contains thousands of lines to search for. I am only using (1) scrapping session. I need to be able to pause between posting data to the http requests. Can someone tell me how to setup a (2) second pause after it is done scrapping each result prior to submitting the next line from the search.csv file? Also, is it possible to have it do a long pause (perhaps ten minutes) after every 1000 lines? My code is below:
Input output error when running from a .bat file
I have a scrape that we normally run from a .bat file on our server and it is working. I need to make some adjustments and when I try to run it locally from a .bat i get the following error in the log file.
An input/output error occurred while connecting to 'https://mydomainname.com/search.cgi'. The message was mydomainname.com.
I get this error anytime i try to run a scrape from outside the UI, it normally works fine from inside the application, but this scrape was not designed to run that way.
I am currently on version 3.0.67a enterprise edition
Thanks
Formatting of Output
Comrades, I am facing a screenscraper problem, that seemeth insurmountable. I hope somebody in this vast void like ether can help me resolve it.
I am trying to harvest data from an events list that looks like this:
Sunday Jan. 20, 2008
- some bandsome venue
- some bandsome venue
- some bandsome venue
... repeated a random #, then restarting
Sunday Jan. 21, 2008
Java Runtime Unexpected Error
Any help debugging this would be appreciated-
I'm running Command-Line SSBasic under XPpro with the latest Java. Quite often the JVM bombs and a log file is generated with the following message below, and the only way I can recover is to reboot:
Date functions
Hello,
Can you tell me what java date functions are available inside of a screen-scraper script.
I am trying to generate a date 2 months in the future like this but it will not take -
//set the expire date
cal = Calendar.getInstance(); // Current date
cal.add(Calendar.MONTH, 2);
Thanks
Scrape: Login - script/extract - Logoff
Forgive it this is a totally newbie question, but, well, I'm a total newbie with screen-scraper. I have gone through most of the tutorials, which helped, but am still in the dark about something.
I need to use SS to log onto a site, request a file for download (session.downloadFile), and then log off the site (by navigating to another URL). If I don't complete the last step the site will be locked for an unspecified amount of time, perhaps preventing a follow-up scrape.
Malformed Request
I'm trying to scrape the following site: https://www.geoaccess.com/wellmark/po/default.asp?SelectedNet=BlueDental....
I'm entering a single zip code, and I keep getting a page with the following message: The page you have requested cannot be displayed because the request for the page was malformed.
Splitting output into multiple files.
Hi all,
First I would like to say thanks to the creators of this software, I love it so far, and once I have gotten it to do what I want I'm definetly going to buy it!
Would someone be so kind to tell me how to add some text to the beginning and the end of the file created? And also how do I split up the file with each file having a 100 records? So it would be Info1, Info2, Info3 each containing 100 records pr. file?
Here's my current code: