Home » Forums » screen-scraper » screen-scraper support for licensed users

Load Session not found in class'org.apache.commons.io.FileUtils

Hi, I've been trying to implement this method: (Save session state)
https://support.screen-scraper.com/documentation/api/session/saveStateToString

By adding the referenced code to a session file after scraping ends which works great.

Then I attempt to retrieve the file before session begins using this method:

https://support.screen-scraper.com/documentation/api/session/loadStateFromString

However I get the error: "not found in class'org.apache.commons.io.FileUtils"

I tried uninstalling and reinstalling according to this:
http://www.screen-scraper.com/blog/2015/04/29/https-connection-issues/

Any idea what the problem might be?

It seems I'm using the latest Java:

=========================================================
=================== Log Variables with Message ===============
screen-scraper Instance Information
=================== Static Values ================
Java Vendor: Oracle Corporation
Java Version: 1.8.0_66
OS Architecture: amd64
OS Name: Windows 10
OS Version: 10.0
Scrape HTTP Client: AsyncNingScrapingHttpClient
SS Connection Timeout: 180 seconds
SS Edition: Enterprise
SS Extractor Timeout: 30000 milliseconds
SS Max Concurrent Scraping Sessions: 5
SS Maximum Memory: 256 MB
SS Memory Use: 12%
SS Run Mode: Workbench
SS Version: 7.0
======== Message logged at: 07/27/2019 11:35:06.508 MDT ========

An error occurred while processing the script: LoadSession
Ali Scrape 1: The error message was: class bsh.EvalError (line 3): FileUtils .readFileToString ( "session_state.xml" , session .getCharacterSet ( ) ) --Sourced file: inline evaluation of: ``import org.apache.commons.io.FileUtils; sessionState = FileUtils.readFileToStri . . . '' : Error in method invocation: Static method readFileToString( java.lang.String, java.lang.String ) not found in class'org.apache.commons.io.FileUtils'

‹ ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver Is there a quick way to getting rid of class="..." strings? ›
allthenamesweretaken on 07/27/2019 at 11:44 am

Starting scraper.Running

I initially got this: (From the downgraded version)

Starting scraper.
Running scraping session: Test - save session
Processing scripts before scraping session begins.
Scraping file: "Single"
Single: Processing scripts before a file is scraped.
Single: Requesting URL: https://support.screen-scraper.com/documentation/api/session/saveStateToString
Single: Processing scripts before all pattern applications.
Processing script: "Modify tags"
ERROR--Test - save session: An error occurred while processing the script: Modify tags
Test - save session: The error message was: class bsh.EvalError (line 4): StringUtils .replacePattern ( s , "
]*>" , "

" ) --Sourced file: inline evaluation of: ``import org.apache.commons.lang3.StringUtils; String s = scrapeableFile.getConte . . . '' : Error in method invocation: Static method replacePattern( java.lang.String, java.lang.String, java.lang.String ) not found in class'org.apache.commons.lang3.StringUtils'
Single: Extracting data for pattern "Test"
Single: The following data elements were found:
Processing scripts always to be run at the end.
Scraping session "Test - save session" finished.

I upgraded back to the unstable version and the test seems to be working fine...

Starting scraper.
Running scraping session: Test - save session
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Scraping file: "Single"
Single: Processing scripts before a file is scraped.
Single: Requesting URL: https://support.screen-scraper.com/documentation/api/session/saveStateToString
Single: Processing scripts before all pattern applications.
Processing script: "Modify tags"
Single: Extracting data for pattern "Test"
Single: The following data elements were found:
Test--DataRecord 0:
CONTENT=~@content@~
Single: Test: Processing scripts after a pattern application.
Single: Test: Processing scripts once if pattern matches.
Single: Test: Processing scripts after all pattern applications.
Single: Processing scripts after a file is scraped.
Processing script: "Save session state"
Saving scrape state to String
Saving cookies
Saving Cookie
--Domain: .screen-scraper.com
--Name: __cfduid
--Value:cookievalue
Saving Cookie
--Domain: .support.screen-scraper.com
--Name: cookie name
--Value:cookesvalue
Processing scripts after scraping session has ended.
Processing scripts always to be run at the end.
Scraping session "Test - save session" finished.

Then I got excited and ran my own

Starting scraper.
Running scraping session: Ali Scrape 1
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Processing script: "LoginScreen"
Scraping file: "LoginScreen"
LoginScreen: Requesting URL: https://passport.login.com/newlogin/account/check.do?fromSite=13&appName=aebuyer
LoginScreen: Processing scripts before all pattern applications.
Processing script: "LoadSession"
ERROR--Ali Scrape 1: An error occurred while processing the script: LoadSession
Ali Scrape 1: The error message was: class bsh.EvalError (line 3): FileUtils .readFileToString ( "session_state.xml" , session .getCharacterSet ( ) ) --Sourced file: inline evaluation of: ``import org.apache.commons.io.FileUtils; sessionState = FileUtils.readFileToStri . . . '' : Error in method invocation: Static method readFileToString( java.lang.String, java.lang.String ) not found in class'org.apache.commons.io.FileUtils'

I also tried the script on a different pc with the exact same result.

 allthenamesweretaken on 07/30/2019 at 9:05 pm

It might be the time that

It might be the time that you're running it. It looks like it's on a file, before the file is scraped. If you change to after the file is scraped?

 jason on 07/31/2019 at 9:18 am

Its not the end of the world

Its not the end of the world if we can't figure it out but it would keep me from pulling my hair our from scraping har files.

I simply replaced your"ModifyTags" file (which doesn't work) with my"LoadSession" which does work.

LoadSession script is copy and pasted from this:
https://support.screen-scraper.com/documentation/api/session/loadStateFromString

Starting scraper.
Running scraping session: Test - save session
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Scraping file: "Single"
Single: Processing scripts before a file is scraped.
Single: Requesting URL: https://support.screen-scraper.com/documentation/api/session/saveStateToString
Single: Processing scripts before all pattern applications.
Single: Extracting data for pattern "Test"
Single: The following data elements were found:
Test--DataRecord 0:
CONTENT=~@CONTENT@~
Single: Test: Processing scripts after a pattern application.
Single: Test: Processing scripts once if pattern matches.
Single: Test: Processing scripts after all pattern applications.
Processing script: "LoadSession"
ERROR--Test - save session: An error occurred while processing the script: LoadSession
Test - save session: The error message was: class bsh.EvalError (line 3): FileUtils .readFileToString ( "session_state.xml" , session .getCharacterSet ( ) ) --Sourced file: inline evaluation of: ``import org.apache.commons.io.FileUtils; sessionState = FileUtils.readFileToStri . . . '' : Error in method invocation: Static method readFileToString( java.lang.String, java.lang.String ) not found in class'org.apache.commons.io.FileUtils'
Processing scripts always to be run at the end.
Scraping session "Test - save session" finished.

Here is my XML file if that helps:

https://docs.google.com/document/d/1UzwGDy8HSw9IE_lRbiMyIy1mHhYP0Le4cr6eNN_-3C0/edit?usp=sharing

Here is a log file according to this thread:
https://support.screen-scraper.com/node/2278

Starting scraper.
Running scraping session: Test instance
Initializing HTTP Client.
Completing scraper initialization.
Processing scripts before scraping session begins.
Processing script: "Write status to log"
=========================================================
=================== Log Variables with Message ===============
screen-scraper Instance Information
=================== Static Values ================
Java Vendor: Oracle Corporation
Java Version: 1.8.0_66
OS Architecture: amd64
OS Name: Windows 10
OS Version: 10.0
Scrape HTTP Client: AsyncScrapingHttpClient2
SS Connection Timeout: 180 seconds
SS Edition: Enterprise
SS Extractor Timeout: 30000 milliseconds
SS Max Concurrent Scraping Sessions: 5
SS Maximum Memory: 4000 MB
SS Memory Use: 8%
SS Run Mode: Workbench
SS Version: 7.0.13a
======== Message logged at: 07/31/2019 13:21:22.578 MDT ========
=========================================================
Scraping file: "IP address"
IP address: Requesting URL: http://www.icanhazip.com
IP address: Extracting data for pattern "Get IP address"
IP address: The following data elements were found:
Get IP address--DataRecord 0:
CURRENT_IP=~@REDACTEDFORSAFENESS@~
Processing scripts after scraping session has ended.
Processing scripts always to be run at the end.
Scraping session "Test instance" finished.

I tried running the script "before session is scraped", "before file is scraped" and "after file is scraped" I also replaced your script from the test session on the extractor pattern to run "before pattern is applied" and "after pattern is applied"

This is going to drive me crazy.

 allthenamesweretaken on 07/31/2019 at 1:30 pm

I see. The readFileToString()

I see. The readFileToString() needs a File object.

import org.apache.commons.io.FileUtils;

// Put the current state in a local variable.
String sessionState = session.saveStateToString( true, true );

// Write the state out to a file.
sutil.writeValueToFile( sessionState, "session_state.xml", session.getCharacterSet() );

File f = new File("session_state.xml");
sessionState = FileUtils.readFileToString(f, session.getCharacterSet());

session.loadStateFromString( sessionState );

 jason on 07/31/2019 at 4:41 pm

I' really sorry...Apparently

I' really sorry...Apparently I need to learn some things. That makes total sense. I was worried it would be something easy and I would be publicly embarrassed.

Thanks so much!

 allthenamesweretaken on 07/31/2019 at 6:09 pm

I just tested on version

I just tested on version 7.0.13a, and it worked. Can you upgrade the screen-scraper version?

  1. Open the workbench and go to options > settings.
  2. Check "allow update to unstable version", and update the maximum memory allocation to 512 (or more)
  3. Save and close.
  4. Click options > check for updates, and run the update.
 jason on 07/29/2019 at 9:30 am

I've upgraded to the unstable

I've upgraded to the unstable version (which looks awesome)

I've increased memory to 4000 mb

I am still getting the same error.

I uninstalled, moved the ss folder and reinstalled
I've upgraded to the unstable version (which looks awesome)
still the same error

I also tried downloading the commons.io package V2.6 from:
http://commons.apache.org/proper/commons-io/

deleted the commons.io located in "C:\Program Files\screen-scraper Enterprise Edition\lib" replaced it with the commons.io v 2.6

still the same error

This will make me seem even less savvy but I did note that in the "import org.apache.commons.io.FileUtils;" statement the "FileUtilis" is not the yellow color as it is with the other classes that I've used for example:
import java.util.Calendar;
the "Calendar" is yellow.

That made me wonder if the java class is not being properly imported however from the error it seems that if "readFileToString" was not found FileUtilsthen it is at least looking in "FileUtils"?

"not found in class'org.apache.commons.io.FileUtils"

Any ideas?

Again, Thanks for your patience and quick responses.

 allthenamesweretaken on 07/29/2019 at 9:25 pm

First, I added the session I

First, I added the session I used to test. It is "Test - save session." Can you download the attachment, test it, and if there is a problem send me a log?

 jason on 07/30/2019 at 10:44 am