Problem with Cookies
SS doesn't seem to be setting a cookie correctly on a site I'm working on.
I hit the front page of the site, and this is the response (part of it, anyway) that I get back:
HTTP/1.1 200 OK
Content-Type: text/html
Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_ssl/2.8.12 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2 mod_perl/1.26
Pragma: No-cache
Content-Language: en-US
Date: Fri, 29 Sep 2006 19:32:25 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Transfer-Encoding: chunked
Set-Cookie: JSESSIONID=0001U0DLVTTMjIkOpoBhM9gcOM-:101gvdp2j
Content-Type: text/html
Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_ssl/2.8.12 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2 mod_perl/1.26
Pragma: No-cache
Content-Language: en-US
Date: Fri, 29 Sep 2006 19:32:25 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Transfer-Encoding: chunked
Set-Cookie: JSESSIONID=0001U0DLVTTMjIkOpoBhM9gcOM-:101gvdp2j
The Set-Cookie header is there.
But when I scrape the next page, the cookie is not sent. Cookie management isn't a pro-only feature is it?
I'd be happy to provide details about the pages scraped by email if necessary.
Thanks!
Adam
Problem with Cookies
Hi Adam,
The quickest way to help you would be for you to simply send me your scraping session so that I can help debug. If that's a possibility, my email address is my first name at screen-scraper.com.
If that's not a possibility, it may be that the site is handling cookies in an unusual way. In your Professional Edition instance, under the "Advanced" tab for your scraping session, you'll notice a few cookie options. Try setting the "Cookie policy" field to "Compatibility" and check the box labeled "Use HTTP strict mode". If that doesn't work, try a few other combinations of different cookie policies with the "strict mode" box checked and un-checked.
Thanks,
Todd
Problem with Cookies
Another follow-up:
I tried going though the proxy with IE instead of FF, with no difference. The scraping still triggered the "Cookies must be enabled" page.
However, I downloaded a copy of the Pro version, switched the HTTP Client to "Internet Explorer" and it worked. Does this shed any light on the situation?
We're in the process of purchasing a copy of SS Pro for development purposes, but the end solution must be compatible with the Basic edition. Is there any way to get around this problem (whatever the problem actually is) without the HTTP Client option?
Problem with Cookies
When I look at the "Last Request" tab for the scrapable file, I do not see a Cookie header there. When I view the entry under the Proxy Session, the request does include the cookie - the same one that was in the Set-Cookie header from the previous response. The domain does not change between the pages - the cookie should be valid.
An intersting thing to note, though: I disabled cookies on IE and went to the site. I was able to browse all the information I need to scrape without accepting cookies - I did not get the "You must enable cookies" page that I get when scraping. When I tried disabling cookies in Firefox, I got the enable cookie page agian.
Hmm. I did use FF for the proxy session. Would this make a difference?
Thanks again for your help!
Adam
Problem with Cookies
Hi Adam,
The cookie management is a feature of both the Basic and Professional editions. The session.setCookie method, however, is only available in the Professional edition. Is that what you're trying to do? Or does it simply not seem to be passing the cookie along? If the latter, you might double-check that the cookie is valid for the domain of the second page.
If none of that helps, feel free to post a reply.
Kind regards,
Todd Wilson