screen-scraper public support

Hello guys. Tim here, ex-employee. I never thought I'd be coming to the forums to ask questions one day :)

So, I've been doing some fixes for a past customer who approached me about working with me directly, and things have been okay. But recently we found that the new login process takes us through the ridiculous roller coaster of redirects and Flash cookie checks.

Is it Possible to import scripts without stopping the server ?
Or is a stop and restart required?

Appreciate - Saurabh

We have a scraping session which works fine on a Windows machine, my development machine.

However when we run the same on our server, which is a Solaris machine we get the following errors

An HTTP error occurred while connecting to 'xxxxxxxxxx'. The message was: Unable to parse header: HTTP/1.0 200 OK.

Any pointers ??

- Dipti

Hi,

I am need to scrape sites with foreign currencies which are in the thousands range ($1,000). The data are then transferred to MySQL database. MySQL has problems with commas (,) and read it as a decimal point. So something that is "100,000" ends up being "100". I am not really familiar with Mysql, so I was wondering is there a way to use Regular Expressions that can scape "$54,321" into "54321"?

Any suggestion is greatly appreciated! Thank you in advance!

Hi we have observed this issue with websites which have changing cookies.

When we capture a scrapable file in the headers we find the cookie set as

ASP.NET_SessionId xokrkt453wnjza45ynqqotfc

I try to capture the same scrapable file after some time and observe that
the value for ASP.NET_SessionId has changed.

Due to this doing a session.setCookie() in my first script to any value does not work,
since the script would work for the time being, but not work after some time.

Working with international characters and I seem to be experiencing the same issues that have befallen many a screen scraper!

Have set the default character set to UTF-8 and the default font to Arial Unicode MS.

The information is being posted to a PHP file which is encoded in UTF-8.

The content type is set to charset=utf-8 in the Accept-Charset: the values are set to: ISO-8859-1, utf-8;q=0.7,*;q=0.7

Have gone through tried Tidy HTML after scraping, no joy.

Hi there,

I'm trying to scrape an ASP shopping site page as my first challenge after going through the tutorials.
I've figured out you need to retain your asp session state thingies throughout so I'm assigning a ~@variable@~ to a portion of scraped data up front, then editing the properties of that variable and setting "save in session variable".
I'm then wanting to use that same variable to post to the next page to keep the asp session alive.

I'm trying to scrape a site which contains cookies and frames.

See script: http://www.blue-curve.com/ASDA%20(Scraping%20Session).sss

Unfortunately the script doesn't seem to take the cookie in and then use it to scrape the content.

The site does however use frames but I've managed to bypass this through the proxy.

Could you advise why I can't see the content?

Hi Friends !

We have a follow error when I use:

session.setCookie("www.paguemenos.com.br", "cep", "01048-000");

Error:

The error message was: The application script threw an exception: java.lang.UnsatisfiedLinkError: com.screenscraper.ie.InternetExplorer.setCookieData(Ljava/lang/String;[B)Z BSF info: null at line: 0 column: columnNo

Could we help me please ?

Dear Sir:

I had followed pages:"http://community.screen-scraper.com/invoking_screen-scraper_from_java" and "http://community.screen-scraper.com/Tutorial_3_page_4_Using_Java" to setup an external java application to get extracted titles of dvd from screen-scraper running in server mode (scraping session by Tutorial 4) on ubuntu desktop 8.04 32bit in WMWare successfully.

Problem occurs while screen-scraper running in server mode on ubuntu server 8.04 64bits.