Home » screen-scraper

screen-scraper support for licensed users

Questions and answers regarding the use of screen-scraper. Only licensed Professional and Enterprise Edition users can post; anyone can read. Licensed users please contact support with your registered email address for access. This forum is monitored closely by screen-scraper staff. Posts are generally responded to in one business day.

Not able to capture Birmingham council

We not able to capture the Birmingham council link from screen scrapper tool

Link is - https://eplanning.birmingham.gov.uk/Northgate/PlanningExplorer/GeneralSearch.aspx

Need your help for the same

Thank you
Barnali

barnali on 08/06/2015 at 4:04 am

how to run a shell script from within a ss script?

When a certain condition is met, I want to run a shell script that I have put in the Screen Scraper main installation directory. So my ss script code would look something like this, inside a while loop:

if (counter == 15)
{
        session.logError("15 iterations reached. Running the shell script and then waiting for 2 minutes");
        [run myshellscript.sh]
        counter = 0;
        sutil.pause(120000);
}
else   
{
        session.logError("Everything fine. Waiting for 10 seconds"
        counter = counter + 1;
        sutil.pause(10000);
}

How can it be done?
Thank you very much,

bogavante on 07/28/2015 at 8:58 am

Screen-Scraper / extractData guru(s)?

Hi all,

Are you an experienced user with decent programming skills and interested in some paid work? If so please take a look ...

Reason for asking is that my coding skills are pretty poor, and I haven't been successful in being able to tweak the extractData API examples to achieve what I'm after.

I have about 20 existing (working) scrapes that don't use that API, and would prefer to keep some new scrapes I need done within Screen-Scraper. That said, I've put one up on elance to see what comes back.

jacko on 07/28/2015 at 7:15 am

Robot Blocking

Hi,

I'm trying to scrape hospital data from the national Blue Cross/Blue Shield site (provider.bcbs.com), and it looks like they're using some kind of bot system to prevent automatic extractions. Has anyone run across anything like this before?

Thanks.

John

jclerie on 07/14/2015 at 7:39 am

Site certificate causing errors

I receive the following error message and both appear to be related to the site certificate:

An input/output error occurred while connecting to 'https://dominionlending.ca/tools/mortgage-rates/'. The message was java.lang.RuntimeException: Could not generate DH keypair.

An input/output error occurred while connecting to 'https://spinmortgage.com/current-rates/'. The message was sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

exdap on 06/29/2015 at 9:35 am

Showing proxy error even proxy is not configured

Hi,

We have screen scrapper server configured on Solaris machine below is the configuration for the same.

Java Vendor : Oracle Corporation
Java Version : 1.7.0_67
OS Architecture : sparc
OS Name : SunOS
OS Version : 5.10
SS Connection Timeout : 1800 seconds
SS Edition : Enterprise
SS Extractor Timeout : 1200000 milliseconds
SS Max Concurrent Scraping Sessions : 500
SS Maximum Memory : 2048 MB
SS Memory Use : 35%
SS Run Mode : Server
SS Version : 6.0.55a

barnali on 06/29/2015 at 4:22 am

More https issues

I am at Java 8 and the latest version of SS 6.1.62a

I had a scrape that used to run and then the site changed to https

An input/output error occurred while connecting to 'https://www.cormacleonard.com/vehicles/?page=1&id=1261%2f'. The message was sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

I have really tried the manual method to get the bits I need through Chrome developer tools, but it simply highlights that I am not a proper developer!

jas777 on 06/18/2015 at 2:39 am

Not sure if sub-extractors is the way for this

Hi, I need to refresh my ScreenScraper knowledge, cause I used it long time ago. Could you tell me what is the best way for scraping this kind of html:

<tr>
<td>Title 1</td>
<td>Amount 1</td>
<td class='catcell'><small>Category 1<br />
Category 2<br />
Category 3<br />
</small></td>
</tr>

<tr>
<td></td>
</tr>

<tr>
<td>Title 2</td>
<td>Amount 2</td>
<td class='catcell'><small>Category 1<br />
Category 2<br />
</small></td>
</tr>

<tr>
<td></td>
</tr>

<tr>
<td>Title 3</td>
<td>Amount 3</td>
<td class='catcell'></td>
</tr>

In order to have:

bogavante on 06/16/2015 at 3:59 am

session.downloadFile

When using the session.downloadFile I receive a message "The server returned a status code of 404". Using the same url in a scrapeableFile returns the binary data without issue.

Thanks, Jeremy

exdap on 06/11/2015 at 11:38 am

Cannot connect to https site with proxy server

When I turn on the proxy server and change the settings in Chrome to the 127.0.0.1 and 8777 I am no longer able to access https sites - only http ones?

I get an "ERR_SSL_VERSION_OR_CIPHER_MISMATCH" message

Have I done something fundamentally wrong?

Site I am trying https://www.rbauction.com

Many Thanks

Jason

jas777 on 06/09/2015 at 5:39 am