Launch Server Headless
Hi,
when running Screen-Scraper server (enterprise edition) on a Mac OS X Leopard or Snow Leopard, the user starting the server has to be logged in and it cannot be started in headless mode. I added the following line
-java.awt.headless=true
to the server.vmoptions file.
Is there a way of launching the server in headless mode?
Thank you,
Edgar
Hi Edgard, Could you give a
Hi Edgard,
Could you give a bit more detail on the behavior you see? I'm assuming that the user account you're using to start screen-scraper in server mode also owns all of the files in the directory where screen-scraper is installed. If that's the case, could you reply back with the output you see when you type the following in the directory where screen-scraper is installed?
./server start
Thanks much,
Todd
Problem solved
Hi Todd,
yes, you're right, the user owns the Screen-Scraper application folder. If I log out from the server and use the Terminal to start up the server using the following command
./server start > message.txt
the message.txt file contains the content:
Starting server
Unrecognized option: -java.awt.headless=true
Could not create the Java virtual machine.
Now, I changed the server.vmoptions file to
-Djava.awt.headless=true
and the server starts up even though the user is logged out. After the start the file server.vmoptions is reset to the previous stage. If I stop the server and restart it again, it still works for Mac OS X Snow Leopard -- obviously, the server runs headless on Snow Leopard without any additional settings.
Unfortunately, on Mac OS X Leopard I get the following output when launching the server
Starting server
Shared archive: sharing disabled for server vm
The result is that I can run the Screen-Scraper Web Admin but after a while the scrapes end saying "Interrupted".
Thank you for your help so far. Do you know what the message "Shared archive: sharing disabled for server vm" means and how I can solve this issue?
Thank you again and kind regards,
Edgar
Hi Edgar, Just to clarify
Hi Edgar,
Just to clarify initially, why are you including "-Djava.awt.headless=true" in the server.vmoptions file? It's possible that's somewhere in our docs, but I don't believe it's needed in this case.
I'm running Leopard, and here's what I did:
1. Turn on "Remote login" under the "Sharing" system preferences applet.
2. Log out of my Mac.
3. From a remote machine log in to my Mac via ssh.
4. Start up screen-scraper with: ./server start
5. Pull up the web interface from the same remote machine.
6. Start a few long scrapes running.
7. Log out of my machine via ssh from the remote server.
8. Monitor the server to ensure that the scrapes continue to run.
I left the scrapes running for about an hour without any problems. I think I'm missing a few pieces of the puzzle, though. Could you walk me more specifically through the steps you're following?
Also, I haven't seen the "Shared archive: sharing disabled for server vm" message before, but will likely research it if I can reproduce the problem you're seeing.
Thanks,
Todd
I do the same
Hi Todd,
these are exactly the steps I'm doing. We have 2 servers Leopard and Snow Leopard.
1. ssh into the server using the user who owns the Screen-Scraper application folder
2. "cd /Applications/screen-scraper\ enterprise\ edition/"
3. "./server start > message_start.out"
The file message_start.out contains the following entries:
Starting server
Shared archive: sharing disabled for server vm
4. I check the status using "./server status"
Shared archive: sharing disabled for server vm
The daemon is running.
5. I use the Monitor to start a scrape
6. The scrape has the status "In Progress" but it's not scraping (I have a lot of logging code which displays the URL of every scraped URL, but it's not showing anything).
7. Eventually the scraping session stops with the status "Completed" (it takes a while though 0.283 hours). Contains nothing but my debugging messages.
I also found the following exception in the "log/stderr.log" file:(Toolkit.java:1530)(Color.java:250)
Exception in thread "DataExtractTimeoutThread" java.lang.InternalError: Can't connect to window server - not enough permissions.
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1824)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1725)
at java.lang.Runtime.loadLibrary0(Runtime.java:822)
at java.lang.System.loadLibrary(System.java:993)
at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)
at java.security.AccessController.doPrivileged(Native Method)
at java.awt.Toolkit.loadLibraries(Toolkit.java:1509)
at java.awt.Toolkit.
at java.awt.Color.
at com.screenscraper.common.DataRecordInfo.setColors(DataRecordInfo.java:100)
at com.screenscraper.scraper.ExtractorPattern$1TimeoutThread.run(ExtractorPattern.java:2035)
Does this have anything to do with the output "Shared archive: sharing disabled for server vm" while starting the server? I don't really know what the "Shared archive" error is but the thread dumb suggest that Screen-Scraper cannot connect to the window server.
Thank you for your help,
Edgar
P.S.: I also get the message (but on the other server). Are these issues connected, I guess not (just for completeness)
An error occurred while executing the SQL statement. The error was Connection is broken: Broken pipe.
java.sql.SQLException: Connection is broken: Broken pipe
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.jdbcStatement.execute(Unknown Source)
at com.screenscraper.util.DataMain.executeInsert(DataMain.java:426)
at com.screenscraper.util.DataMain.executeInsert(DataMain.java:454)
at com.screenscraper.data.DScrapingSession.setScheduledScrapingSession(DScrapingSession.java:969)
at com.screenscraper.business.BScrapingSession.setScheduledScrapingSession(BScrapingSession.java:686)
at com.screenscraper.scraper.ScheduledScrapingSession.save(ScheduledScrapingSession.java:444)
at com.screenscraper.server.ScrapingSessionServiceImpl.disableEnableScheduledScrapingSession(ScrapingSessionServiceImpl.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:562)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:188)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:224)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at com.screenscraper.web.ServletFilter.doFilter(ServletFilter.java:74)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2416)
at com.gefionsoftware.lws.core.StandardHostValve.invoke(StandardHostValve.java:156)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:601)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:392)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:572)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:637)
SQL was: INSERT INTO scheduled_scraping_session ( scraping_sessionid, timeout, time_to_run, frequency, relative_time_threshold, relative_records_scraped_threshold,settable_session_variables,enabled) VALUES ( 1799, -1, 1273744800000, 86400, -1, -1, 'USERNAME=west2680&PASSWORD=WtEEnEqE', 0)
Hi Edgard, We've just tried
Hi Edgard,
We've just tried this between a machine running Leopard and Snow Leopard (with a 64-bit chip), and still can't seem to be able to reproduce the problem. Are you running Mac OS X Server, by chance?
To clarify a bit further, we went through the same steps you give above, but never see this message: "Shared archive: sharing disabled for server vm."
The Snow Leopard machine reports the following Java version:
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04-248-10M3025)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01-101, mixed mode)
The error you report that contains, "Can't connect to window server - not enough permissions." has been fixed in version in version 4.5.59a of screen-scraper, though I don't think that's the root of the problem.
The SQL exception you report would probably be a result of the fact that screen-scraper can't spawn the database process, likely because of the very same "Shared archive: sharing disabled for server vm." issue.
Also, this doesn't appear to be an issue that's necessarily inherent to screen-scraper itself, but seems instead to be an issue with Java in general. Are you able to run other Java apps on that machine in this same scenario without problems? For example, can you start up Tomcat or something similar?
As another test you might also try running screen-scraper in interactive mode, which allows you to invoke the Java class directly without going through the server daemon. You would do that with the following command:
java -Xmx128M -jar screen-scraper.jar --start-server --interactive
Lastly, in all of this testing it's very possible that your screen-scraper database has become corrupted. If you log into the machine are you able to start up the screen-scraper workbench normally?
Thanks,
Todd