Home » Forums » screen-scraper » screen-scraper support for licensed users

HELP! Can't Call From Command Line After Upgrading to 7.0.14a

Under Windows command line I would call a scraping session from here:
"C:\Program Files\screen-scraper Professional Edition\jre\bin\java" -jar screen-scraper.jar -s "SessionNameHere"
I have just upgraded and none of my scraping sessions will run (indeed that path no longer even seems to exist)
Has the upgrade failed or do I need a new command line?
Thanks as always
Jason

After rebooting this seems to have worked itself out, however the new version seems much slower than the old. I have a few scrapes that now give me a 502 error code as well which didn't happen prior to upgrade?

‹ Easy way to get around Google’s captcha? Creating header records ›
Jason Truckpages on 11/29/2022 at 6:52 am

CURL shipped with Windows 10

I already use CURL - it is in Windows 10 - however, I cannot get it to work with SS. Should I download/install a different version for the SS directory?

It is interesting that these scrapes all worked before the upgrade? It can't be that the sites are too 'modern' there must be another issue?

 Jason Truckpages on 12/19/2022 at 9:13 am

Can I get the log I requested

Can I get the log I requested elsewhere in this thread?

I know SS works in Windows 11, so that shouldn't be the hard part.

 jason on 12/19/2022 at 4:33 pm

Sorry - missed the request

Starting scraper.
Running scraping session: Kellands ss
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Processing script: "Kellands ss"
=========================================================
=================== Log Variables with Message ===============
screen-scraper Instance Information
=================== Static Values ================
Java Vendor: Oracle Corporation
Java Version: 1.8.0_66
OS Architecture: amd64
OS Name: Windows 10
OS Version: 10.0
Scrape HTTP Client: AsyncScrapingHttpClient2
SS Connection Timeout: 180 seconds
SS Edition: Professional
SS Extractor Timeout: 30000 milliseconds
SS Max Concurrent Scraping Sessions: 5
SS Maximum Memory: 256 MB
SS Memory Use: 15%
SS Run Mode: Workbench
SS Version: 7.0.14a
======== Message logged at: 12/21/2022 18:01:57.517 GMT ========
=========================================================
Scraping file: "New Scrapeable File"
New Scrapeable File: Requesting URL: https://kellands.com/product-category/new/doosan/wheeled-excavators/
New Scrapeable File: Warning! Received a status code of: 502.
Processing scripts after scraping session has ended.
Processing scripts always to be run at the end.
Scraping session "Kellands ss" finished.

 Jason Truckpages on 12/21/2022 at 12:02 pm

From the log, I see you are

From the log, I see you are using the Java that was bundled with screen-scraper.

Have you installed Java 18? If you open a command line, and type in

java -version

What does it show?

 jason on 12/27/2022 at 3:18 pm

Java version

java version "19.0.1" 2022-10-18
Java(TM) SE Runtime Environment (build 19.0.1+10-21)
Java HotSpot(TM) 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)

 Jason Truckpages on 01/05/2023 at 3:58 am

Okay, so if instead of

Okay, so if instead of clicking the shortcut or EXE to launch screen-scraper:

  1. Make sure screen-scraper is closed
  2. Open a CMD window, and go to the screen-scraper install directory. Usually it's \PROGRAM FILES\SCREEN-SCRAPER ENTERPRISE EDITION\
  3. Run the command: java -jar screen-scraper.jar

Then if you run the same scrape it should confirm you're running in Java 19, and then try your other scrapes.

 jason on 01/05/2023 at 10:19 am

Running V19 - Still 502 errors (see log)

Starting scraper.
Running scraping session: Kellands ss
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Processing script: "Kellands ss"
=========================================================
=================== Log Variables with Message ===============
screen-scraper Instance Information
=================== Static Values ================
Java Vendor: Oracle Corporation
Java Version: 19.0.1
OS Architecture: amd64
OS Name: Windows 10
OS Version: 10.0
Scrape HTTP Client: AsyncScrapingHttpClient2
SS Connection Timeout: 180 seconds
SS Edition: Professional
SS Extractor Timeout: 30000 milliseconds
SS Max Concurrent Scraping Sessions: 5
SS Maximum Memory: 256 MB
SS Memory Use: 6%
SS Run Mode: Workbench
SS Version: 7.0.14a
======== Message logged at: 01/09/2023 11:25:18.614 GMT ========
=========================================================
Scraping file: "New Scrapeable File"
New Scrapeable File: Requesting URL: https://kellands.com/product-category/new/doosan/wheeled-excavators/
New Scrapeable File: Warning! Received a status code of: 502.
Processing scripts after scraping session has ended.
Processing scripts always to be run at the end.
Scraping session "Kellands ss" finished.

 Jason Truckpages on 01/09/2023 at 5:27 am

I have tried them all but no luck!

They all give the sale 502 error apart from the CURL but this does not actually work as far as the scrapes are concerned...

 Jason Truckpages on 12/06/2022 at 4:29 am

I attached a session with the

I attached a session with the name "Kellands ss"

Can you run it and send me the log?

Starting scraper.
Running scraping session: Kellands ss
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Scraping file: "New Scrapeable File"
New Scrapeable File: Requesting URL: https://kellands.com/product-category/new/doosan/wheeled-excavators/
Processing scripts after scraping session has ended.
Processing scripts always to be run at the end.
Scraping session "Kellands ss" finished.
Starting scraper.
Running scraping session: Kellands ss
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Processing script: "Kellands ss"
=========================================================
=================== Log Variables with Message ===============
screen-scraper Instance Information
=================== Static Values ================
Java Vendor: Oracle Corporation
Java Version: 18.0.1.1
OS Architecture: amd64
OS Name: Windows 11
OS Version: 10.0
Scrape HTTP Client: Async Http Client v2
SS Build: 2022.145.2013
SS Connection Timeout: 30 seconds
SS Disk Use: 22.185%
SS Edition: Enterprise
SS Extractor Timeout: 30000 milliseconds
SS Max Concurrent Scraping Sessions: 10
SS Maximum Memory: 1024 MB
SS Memory Use: 19%
SS Run Mode: Workbench
SS Version: 7.0.14a
======== Message logged at: 12/06/2022 13:33:41.532 MST ========
=========================================================
Scraping file: "New Scrapeable File"
New Scrapeable File: Requesting URL: https://kellands.com/product-category/new/doosan/wheeled-excavators/
Processing scripts after scraping session has ended.
Processing scripts always to be run at the end.
Scraping session "Kellands ss" finished.

 jason on 12/06/2022 at 2:37 pm

There are at least two that were working but now error 502

https://kellands.com/product-category/new/doosan/wheeled-excavators/
https://www.junction6.com/vcategory/tractor-units/page/1

There may be others, but these are the tow that came to light from last night's jobs.

I could roll back to the earlier version, but I would rather make this work.

Thanks
Jason

 Jason Truckpages on 11/30/2022 at 6:25 am

Version 7.0 used Java 8, but

Version 7.0 used Java 8, but 7.14a can use a newer one. If you can install JRE 11, and then in Windows 11, to use the system's Java instead of the bundled one:

  1. Install the newer version of Java
  2. Find the path to that Java. We need the path to the directory that contains the bin subdirectory. Example: C:\Program Files\Java\jdk-[your version]\
  3. In the screen-scraper installation directory, rename the jre subdirectory to jre.old
  4. Open a command line, and navigate to the screen-scraper installation directory
    1. Create a symbolic link to the system's Java directory. So for the example above you would: mklink /d jre 'C:\Program Files\Java\jdk-[your version]'
    2. This will make a new folder named jre, but it's a link to the Java install.
    3. Start screen-scraper to make sure it works.
  5. Once you are convinced screen-scraper is working, you can delete the jre.old.
 jason on 12/01/2022 at 10:53 am

Upgraded but still not working

Thanks for your help here.

I have carried out your instructions, Screen scraper is working after I changed to jdk-19.

However the scrapes still show a 502 when attempting to access a (growing) list of 5 sites

Is there something about cookies or headers that maybe needs to be set for these sites in the newer version of SS?

I have this new java error message though that I wasn't getting before:
java.lang.NoClassDefFoundError: com/sun/net/ssl/internal/ssl/Provider
at com.screenscraper.view.ViewMainFrame.displayStartPage(ViewMainFrame.java:1023)
at com.screenscraper.view.ViewMainFrame.init(ViewMainFrame.java:930)
at com.screenscraper.controller.ControllerMain.init(ControllerMain.java:223)
at com.screenscraper.controller.ControllerMain.(ControllerMain.java:90)
at com.screenscraper.controller.ControllerMain.launchWorkbench(ControllerMain.java:796)
at com.screenscraper.controller.ControllerMain.main(ControllerMain.java:762)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
at com.exe4j.runtime.WinLauncher.main(Unknown Source)
at com.install4j.runtime.launcher.WinLauncher.main(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.sun.net.ssl.internal.ssl.Provider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 11 more

 Jason Truckpages on 12/02/2022 at 4:58 am

Now on the scraping session >

Now on the scraping session > advanced tab you have some choices of HTTP Client. You should try each of those to see if it helps.

Even version 7.14a is pretty old, but since I mostly scrape data more than sell software, I have newer versions that should work, so if this won't then let me know and we'll have to see about getting it all to you.

 jason on 12/05/2022 at 2:42 pm

Still not working for some

They all give the sale 502 error apart from the CURL but this does not actually work as far as the scrapes are concerned...

Seems odd that it is just a number of sites?

 Jason Truckpages on 12/07/2022 at 4:38 am

You could set up cURL with

You could set up cURL with the instructions here: https://support.screen-scraper.com/node/2554

A lot of sites and web servers are getting update beyond the point screen-scraper version 7 is. We have been keeping up to scape stuff, but haven't made the coordinated effort to release it.

 jason on 12/07/2022 at 3:26 pm