screen-scraper and linux-vserver

We're trying to run screen-scraper enterprise 5.5 (latest update) on a linux-vserver guest and we're getting errors (it seems) because the screen-scraper server (or HSQLDB, or both) are trying to bind and connect to 127.0.0.1. Is there a configuration directive to control this? Can I tell it to use the vserver guest accessible interface? We're open to any approach to solve this one.

This page: http://linux-vserver.org/Problematic_Programs talks about several applications that use hardcoded references to localhost as 127.0.0.1 (which is what we think is happening with screen-scraper).

Thanks!

-Tom

tmonaco on 06/03/2011 at 10:29 am

screen-scraper support for licensed users

screen-scraper on linux

I start screen scraper server from SHELL SCRIPT using this command

"jre/bin/java -Xmx128M -jar screen-scraper.jar --start-server --interactive"

and I get following message in the shell as follows

"Initializing screen-scraper server. Please wait...
Ready.
Type "quit" to quit.
? Warning: You're not using a Java Development Kit (SDK) environment so JSP pages can not be compiled.
Starting service LiteWebServer Base Module (lws-base/3.0.3) (with LiteWebServer JSP Module (lws-jsp/1.1.2))"

Now, When i run web scraper in Mozilla browser using http://127.0.0.1:8779 and run my scrapes.. i keep getting errors in SHELL SCRIPT WINDOW which says

"// Error: Attempt to access method on primitive... allowing bsh.Primitive to peek through for debugging"

Shell window keeps on adding that line and finally it kills the server.

Any idea why it is doing so. I want to scrape huge amount of data and also need to create log files. Please help.

sunshineuser on 08/16/2011 at 4:18 am

Error log contents

Here's the error log:

An error occurred while executing the SQL statement. The error was socket creation error.
java.sql.SQLException: socket creation error
        at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
        at org.hsqldb.jdbc.jdbcConnection.(Unknown Source)
        at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
        at org.hsqldb.jdbcDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:525)
        at java.sql.DriverManager.getConnection(DriverManager.java:171)
        at com.screenscraper.util.DataMain.getConnection(DataMain.java:101)
        at com.screenscraper.util.DataMain.executeSelect(DataMain.java:347)
        at com.screenscraper.util.DataMain.executeSelect(DataMain.java:373)
        at com.screenscraper.data.DScrapingSession.getOrphanExtractorPatterns(DScrapingSession.java:865)
        at com.screenscraper.business.BScrapingSession.purgeOrphanExtractorPatterns(BScrapingSession.java:806)
        at com.screenscraper.controller.ControllerMain.main(ControllerMain.java:514)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
        at com.install4j.runtime.Launcher.main(Unknown Source)
Sorry, screen-scraper is unable to connect to its internal database. Please see this page for help: http://community.screen-scraper.com/faq/CorruptedDatabase

Thinking that the database wasn't up correctly, I ran netstat, and sure enough, it's listening on 9003/tcp...the obscured ip is a
192.168.xxx/24, and is the internal network interface for this vserver.

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 [OBSCURED]:9003           0.0.0.0:*                   LISTEN      8636/java

The database is not likely corrupted - I get this on a fresh install with a brand new db - and using a copy of a known working db from another machine doesn't work either.

We're supposing what's happening is that the DB is listening on the correct adapter, however SS is trying to connect to 127.0.0.1. As we've seen with a few other apps that try to do that on a vserver guest, it fails.

tmonaco on 06/10/2011 at 10:17 am

Tom, Do ps ax | grep java and

Tom,

ps ax | grep java

and kill any processes belonging to screen-scraper. Then, try running screen-scraper.

-Scott

swilsonmc on 06/10/2011 at 11:38 am

That didn't help - here's how

That didn't help - here's how I tested it:

From within the screen-scraper install directory (running 5.5.9a, after updating this AM)...

1. checked for running java processes -> none from screen-scraper are present.
2. started the server -> no errors from that command
3. check server status -> daemon is stopped.
4. cat log/error.log:

-bash-3.2$ cat log/error.log 
An error occurred while executing the SQL statement. The error was socket creation error.
java.sql.SQLException: socket creation error
	at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
	at org.hsqldb.jdbc.jdbcConnection.(Unknown Source)
	at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
	at org.hsqldb.jdbcDriver.connect(Unknown Source)
	at java.sql.DriverManager.getConnection(DriverManager.java:525)
	at java.sql.DriverManager.getConnection(DriverManager.java:171)
	at com.screenscraper.util.DataMain.getConnection(DataMain.java:101)
	at com.screenscraper.util.DataMain.executeSelect(DataMain.java:347)
	at com.screenscraper.util.DataMain.executeSelect(DataMain.java:373)
	at com.screenscraper.data.DScrapingSession.getOrphanExtractorPatterns(DScrapingSession.java:865)
	at com.screenscraper.business.BScrapingSession.purgeOrphanExtractorPatterns(BScrapingSession.java:806)
	at com.screenscraper.controller.ControllerMain.main(ControllerMain.java:514)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
	at com.install4j.runtime.Launcher.main(Unknown Source)
Sorry, screen-scraper is unable to connect to its internal database. Please see this page for help: http://community.screen-scraper.com/faq/CorruptedDatabase
An error occurred while executing the SQL statement. The error was socket creation error.
java.sql.SQLException: socket creation error
	at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
	at org.hsqldb.jdbc.jdbcConnection.(Unknown Source)
	at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
	at org.hsqldb.jdbcDriver.connect(Unknown Source)
	at java.sql.DriverManager.getConnection(DriverManager.java:525)
	at java.sql.DriverManager.getConnection(DriverManager.java:171)
	at com.screenscraper.util.DataMain.getConnection(DataMain.java:101)
	at com.screenscraper.util.DataMain.executeSelect(DataMain.java:347)
	at com.screenscraper.util.DataMain.executeSelect(DataMain.java:373)
	at com.screenscraper.data.DScrapingSession.getLE(DScrapingSession.java:1330)
	at com.screenscraper.business.BScrapingSession.getLE(BScrapingSession.java:889)
	at com.screenscraper.util.General.ph(General.java:3981)
	at com.screenscraper.util.General$1TeleThread.run(General.java:3937)

5. check for running screen-scraper processes:

31120 pts/19   Sl     0:00 /usr/java/jdk1.5.0_07/jre/bin/java -Xmx64M -classpath /home/scraperuser/screen-scraper_enterprise/.install4j/i4jruntime.jar:/home/scraperuser/screen-scraper_enterprise/screen-scraper.jar com.screenscraper.util.StartHypersonic 9003

6. killed the above process.
7. deleted the logs again.
8. server start.
9. server status -> daemon is down.
10. cat log/error.log -> same error as above.

tmonaco on 06/13/2011 at 9:29 am

Tom, Have a look at the

Tom,

Have a look at the following...

http://community.screen-scraper.com/faq/CorruptedDatabase

...if the problem persists please post the command you're using to start screen-scraper's server and we'll take a look.

-Scott

swilsonmc on 06/13/2011 at 12:14 pm

The database isn't corrupted.

The database isn't corrupted. I'm using the server script included with screen-scraper 5.5 to start the server. The command I use is

./server start

from the installation directory.

As I've said, this problem ONLY occurs when running on a linux-vserver guest.

tmonaco on 06/13/2011 at 1:15 pm

Tom, I'm not sure what to

Tom,

I'm not sure what to make of this. I've asked our lead developer to have a look.

We'll let you know what we find.

-Scott

swilsonmc on 06/15/2011 at 2:37 pm

Tom, Can you provide any

Tom,

Can you provide any error messages being generated by screen-scraper? Messages appearing either during installation or from any one of the logs screen-scraper generates. All log files can be found in screen-scraper's installation directory.

Thanks,
Scott

swilsonmc on 06/08/2011 at 12:54 pm

Hi Tom, screen-scraper

Hi Tom,

screen-scraper actually connects using "localhost". In working with a virtual server recently, perhaps similar to the one you're using, I found that the issue was actually with an incompatibility between the Java Runtime Environment that ships with screen-scraper and the Linux environment in which it was running. The solution was for me to download and install a JRE directly from Oracle, then direct screen-scraper to use that JRE instead. The FAQ located here:

http://community.screen-scraper.com/FAQ/BSD

Provides instructions on how to set all of that up. Would you mind giving that a try, then posting back with the outcome?

Thanks,

Todd Wilson

todd on 06/20/2011 at 8:55 am

Still getting the same

Still getting the same error...

I downloaded jre1.6.0_26 from oracle, unpacked it into the screen-scraper_enterprise directory.
Set paths in the server and screen-scraper scripts per the link.
Didn't work.
Moved jre directory to originaljre, and symlinked jre -> jre1.6.0_26.
Still didin't work.

It's evident from the output of ps that the correct runtime is being used.

tmonaco on 07/06/2011 at 4:59 pm

I'll post back once I get

I'll post back once I get some time. I'm currently on a higher-priority project, so it might be a few days.

Thanks! :)

tmonaco on 06/23/2011 at 2:37 pm

Hi, One other possibility

Hi,

One other possibility occurs to me--we've had varying success running screen-scraper in "virtual dedicated" hosting environments. This is where the hosting provider provisions you a portion of a machine's CPU, RAM, etc. We've seen cases where the hosting provider claims to place no restrictions on what can be run, but does, actually, limit processes that can be spawned. Could it be that you're operating in such an environment? If so, you might want to do some testing on the JRE itself. For example, try running a simple Java program or something like that. When screen-scraper starts up it spawns a second process that runs its database; it's possible that your hosting environment is inhibiting that. As another possibility for finding clues, you might check the "error.log" (and possibly "stderr.log") file(s) found in screen-scraper's "log" folder. That might give clues as to what's interfering.

Todd

todd on 07/08/2011 at 2:46 pm

More testing results

I looked into the limits issue - it looks like it's spawning the db process just fine - in fact, I have to manually SIGINT the database proc to get it to quit - the server stop script doesn't do anything to it once the server's main process dies for failure to connect to the db.

In the meantime, I tried a few other things:

I dropped the latest version of hsqldb.jar (2.2.5) in lib and overwrote the jar that was in there, and now I'm getting more useful errors instead of a catch-all socket create failure message.

An error occurred while executing the SQL statement. The error was java.net.ConnectException: Connection refused.
java.sql.SQLTransientConnectionException: java.net.ConnectException: Connection refused
        at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
        at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
        at org.hsqldb.jdbc.JDBCConnection.(Unknown Source)
        at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
        at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(Unknown Source)
        at java.sql.DriverManager.getConnection(Unknown Source)
        at com.screenscraper.util.DataMain.getConnection(DataMain.java:101)
        at com.screenscraper.util.DataMain.executeSelect(DataMain.java:347)
        at com.screenscraper.util.DataMain.executeSelect(DataMain.java:373)
        at com.screenscraper.data.DScrapingSession.getOrphanExtractorPatterns(DScrapingSession.java:865)
        at com.screenscraper.business.BScrapingSession.purgeOrphanExtractorPatterns(BScrapingSession.java:806)
        at com.screenscraper.controller.ControllerMain.main(ControllerMain.java:514)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
        at com.install4j.runtime.Launcher.main(Unknown Source)
Caused by: org.hsqldb.HsqlException: java.net.ConnectException: Connection refused
        at org.hsqldb.ClientConnection.openConnection(Unknown Source)
        at org.hsqldb.ClientConnection.initConnection(Unknown Source)
        at org.hsqldb.ClientConnection.(Unknown Source)
        ... 17 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(Unknown Source)
        at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.PlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at java.net.Socket.(Unknown Source)
        at java.net.Socket.(Unknown Source)
        at org.hsqldb.server.HsqlSocketFactory.createSocket(Unknown Source)
        ... 20 more

I'm also trying to reproduce the problem outside of screen-scraper with a simple class to connect to the database - what connection string is being used to connect to the local hsql db? I can't seem to make this part of it even work on my mac (SS works fine on my mac, server and gui, by the way) either, so I'm thinking I've got to be doing something wrong on that one.

I also tried changing the DB port in resource/conf/screen-scraper.properties - but that was not successful in fixing the connection failure.

Thanks!

-Tom

tmonaco on 07/11/2011 at 2:59 pm

Hi, You can use this

Hi,

You can use this connection string to connect to screen-scraper's database:

jdbc:hsqldb:hsql://localhost:9003/ss_server

This assumes, of course, that the database process is already running, and that port 9003 has been designated.

Todd

todd on 07/12/2011 at 4:36 pm

more testing results...

OK, I think we have some progress here...

I have this simple app:

import java.sql.Connection;
import java.sql.DriverManager;

public class VserverHsqlTest {

	public static void main(String[] args) throws Exception {
	
		if( args.length != 1) {
			System.out.println("Args: connectionString");
			System.exit(1);
		}
	
            Class.forName("org.hsqldb.jdbcDriver" );
	    Connection c = DriverManager.getConnection(args[0],"", "");
	}

}

As you can see, it takes the connection string as it's only command line param. Here's where it gets interesting...

Using java -cp .:hsqldb.jar VserverHsqlTest jdbc:hsqldb:hsql://localhost:9003/ss_server, I get...

Exception in thread "main" java.sql.SQLTransientConnectionException: java.net.ConnectException: Connection refused
	at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
	at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
	at org.hsqldb.jdbc.JDBCConnection.(Unknown Source)
	at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
	at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at VserverHsqlTest.main(VserverHsqlTest.java:15)
Caused by: org.hsqldb.HsqlException: java.net.ConnectException: Connection refused
	at org.hsqldb.ClientConnection.openConnection(Unknown Source)
	at org.hsqldb.ClientConnection.initConnection(Unknown Source)
	at org.hsqldb.ClientConnection.(Unknown Source)
	... 6 more

Using java -cp .:hsqldb.jar VserverHsqlTest jdbc:hsqldb:hsql://127.0.0.1:9003/ss_server gets me the same error.

The cool part - if I use the vserver context's correct IP, as seen here...
java -cp .:hsqldb.jar VserverHsqlTest jdbc:hsqldb:hsql://192.168.x.y:9003/ss_server

Exception in thread "main" java.sql.SQLInvalidAuthorizationSpecException: invalid authorization specification - not found: 
	at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
	at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
	at org.hsqldb.jdbc.JDBCConnection.(Unknown Source)
	at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
	at org.hsqldb.jdbc.JDBCDriver.connect(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at VserverHsqlTest.main(VserverHsqlTest.java:15)
Caused by: org.hsqldb.HsqlException: invalid authorization specification - not found: 
	at org.hsqldb.error.Error.error(Unknown Source)
	at org.hsqldb.ClientConnection.(Unknown Source)
	... 6 more

So, using the context's actual ip address (instead of "localhost" or "127.0.0.1") results in it giving an authentication error, but demonstrates it can connect. If there was a way to specify the hostname part of the connection string via the .properties, it should fix the connection issue.

Thoughts?

Thanks,
Tom

tmonaco on 07/13/2011 at 9:32 am

Hi Tom, We just released

Hi Tom,

We just released version 5.5.16a of screen-scraper, which allows you to set screen-scraper's database host via the "DatabaseHost" property. After upgrading, just add something like this to your screen-scraper.properties file (found in the "resource/conf" folder):

DatabaseHost=192.168.1.123

Would you mind giving that a shot and letting us know if it works?

Thanks,

Todd

todd on 08/19/2011 at 5:00 pm

I'll test this as soon as I

I'll test this as soon as I can.

Thanks!

tmonaco on 08/25/2011 at 10:47 am

Search

Community

screen-scraper

User login

screen-scraper and linux-vserver

screen-scraper on linux

Error log contents

Tom, Do ps ax | grep java and

That didn't help - here's how

Tom, Have a look at the

The database isn't corrupted.

Tom, I'm not sure what to

Tom, Can you provide any

Hi Tom, screen-scraper

Still getting the same

I'll post back once I get

Hi, One other possibility

More testing results

Hi, You can use this

more testing results...

Hi Tom, We just released

I'll test this as soon as I