Script not found error on headless machine
I am trying to run screen-scraper on a headless (no GUI) version of Gentoo Linux with Java 1.5 (although I guess that doesn't matter since ss already has a JRE). I am using the Hello World example from the downloaded hello_world1.zip file. I changed the name of the interpreted java file from Hello World (Scraping Session).xml to Hello World.xml. I have also tried renaming it to HelloWorld.xml (no spaces) and changed the command line appropriately.
For the command line I use the following.
jre/bin/java -jar screen-scraper.jar --run-script "Hello World" --params "TEXT_TO_SUBMIT=Hello+World"
The response is:
Initializing screen-scraper...
The script "Hello World" was not found. Please check the name and try again.
I have made several variations on the text in "Hello World", from "Hello World.xml" to "HelloWorld" to "HelloWorld.xml" (the last 2 I of course changed the filename as well). I cannot run this example on DOS like this either.
However, I can run the example from DOS when I substitute "Hello World--begin scraping session". Note that I have run through the tutorials so that the script by the name of "Hello World--begin scraping session" exists because of creating it during the tutorial. But how do I make this script work on a headless Linux where I can't go through the tutorial first?
Thanks for any feedback.
Script not found error on headless machine
Hi,
Try this FAQ for help on that:
[url]http://www.screen-scraper.com/support/faq/faq.php#GUILessUpdate[/url]
Todd
Script not found error on headless machine
Ok, so we need to upgrade. Two questions:
1. What is the easiest method for a headless linux upgrade? It seems most instructions are of the point-n-click variety. It should also be noted that we're not concerned with backups at this time.
2. Will we need to modify AND / OR reimport the XML files after the upgrade?
Thanks.
Script not found error on headless machine
Hi,
Sorry for leading you into wayward paths of darkness. After some investigating, I found out that this was a bug the resulted from a new feature we added in version 3.0. The -w switch isn't something we publish in our public documentation, so it didn't make it into our normal testing regimen.
Fortunately, we've provided a fix: [url]http://blog.screen-scraper.com/2007/02/13/version-303a-of-screen-scraper-available/[/url].
Note that one other modification you'll want to make is in the resource/conf/log4j.properties file. Change this line:
log4j.appender.file.File=log/error.logTo have an absolute path, like this:
log4j.appender.file.File=/usr/local/screen-scraper/log/error.logIt also occurred to me that, as an alternative to all of this, you could simply write a shell script that looks something like this:
jre/bin/java -jar screen-scraper.jar -s "My Script"
Todd
Script not found error on headless machine
Yep, definitely need to find a way to apply that -w flag. After taking a minute to look over the failure output in my previous post, I decided to test the -w flag by replicating the /log subdirectory in my current test launch location. And while it still fails to run, it provides a different error than before:
Exception in thread "main" java.lang.NullPointerException
at com.screenscraper.util.General.queueCleanUpCommandLineLockFiles(General.java:1799)
at com.screenscraper.controller.ControllerMain.main(ControllerMain.java:508)
Alas, nothing actually got written to my phony log files (nor the legitmate ones located in /usr/local/screen-scraper/log/) so I am unable to dig deeper, but hopefully this minor test stands as sufficient proof that the current usage of the -w location flag is not working.
Script not found error on headless machine
Todd,
Let me begin by apologizing for the delay in resuming troubleshooting as I've been out of the office. the addition of the -w flag appears to be ineffective:
[color=green]Success[/color] (normal run from working directory):
Initializing screen-scraper...
Running script "rev_lookup_launch"...
Starting scraper.
Running scraping session: rev_lookup_scrape
Processing scripts before scraping session begins.
Scraping file: "rev_lookup"
...
[color=green]Success[/color] (adding -w flag from working directory):
Initializing screen-scraper...
Running script "rev_lookup_launch"...
Starting scraper.
Running scraping session: rev_lookup_scrape
Processing scripts before scraping session begins.
Scraping file: "rev_lookup"
...
[color=red]Failure[/color] (adding -w flag from non-working directory):
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: log/error.log (No such file or directory)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:177)
at java.io.FileOutputStream.<init>(FileOutputStream.java:102)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:272)
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:156)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:151)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:247)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:123)
at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:87)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:645)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:603)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:500)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:406)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:307)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:315)
at com.screenscraper.controller.ControllerMain.main(ControllerMain.java:471)
Exception in thread "main" java.lang.NullPointerException
at com.screenscraper.util.General.queueCleanUpCommandLineLockFiles(General.java:1799)
at com.screenscraper.controller.ControllerMain.main(ControllerMain.java:508)
root@foo:/home/test#
Note that I did try it with a trailing /, as in: -w "/usr/local/screen-scraper/" and that didn't work, either.
This is our only remaining hurdle...so close, yet so far...
;)
Script not found error on headless machine
Woop! My fault. Try this:
/usr/local/screen-scraper/jre/bin/java -jar /usr/local/screen-scraper/screen-scraper.jar -s "My Script" -w "/usr/local/screen-scraper"Todd
Script not found error on headless machine
Even with the suggested changes, if we try to run it from any location other than the install directory it has issues:
Unable to access jarfile screen-scraper.jarWe're on Ubuntu, if that matters.
Script not found error on headless machine
Hi,
Sounds like you're making great progress. You can also use a -w command line switch to tell screen-scraper the working directory it should use. For example, you might use a shell script with this:
/usr/local/screen-scraper/jre/bin/java -jar screen-scraper.jar -s "My Script" -w "/usr/local/screen-scraper"You should be able to run that script from any directory on the file system.
Todd
Script not found error on headless machine
That was easy. Works as it should. Thanks for the patience and the help.
One last item that we are having trouble with is invoking the command from anywhere outside of /usr/local/screen-scraper, which could introduce some complications when calling it from inside an unrelated script.
Any suggestions?
Script not found error on headless machine
Yep, I was doing something stupid for sure.
I have an XML for the scrape, with all the necessary extraction information. And I have an XML for the script that writes the extracted data to a file. What I was missing was a script to run the scrape.
I'm off to go see if I can figure out how to make one of those.
Script not found error on headless machine
Hi,
Looks like we're making ground. I don't think I'll need to see the XML files. My guess is that you're invoking the "log" method on the RunnableScrapingSession object, which won't work. You can only invoke the "log" method on the "session" object, once a scraping session is running. Generally start scripts will be very small, and will simply create a RunnableScrapingSession, optionally set a few session variables, then start it scraping. There should rarely be any need to log anything. Try modifying your script to get rid of those "log" calls, and hopefully it will start the scraping session.
Each time you import the files get over-written, so you shouldn't need to clean anything up. If you want, though, you could always copy over database files from a clean instance of screen-scraper. The database files consist of the ss.* files found in the "resource/db" folder. Just be sure that both instances of screen-scraper are not running when you copy anything over.
Kind regards,
Todd
Script not found error on headless machine
Gotcha. We're a lot closer now but it might help for me to post the two xml files contents?
Since I have been running this over and over, how do I purge imported data to start with a clean import?
Anyway, here's what we get now:
Initializing screen-scraper...
Running script "rev_lookup_script"...
An error occurred while processing the script: rev_lookup_script.
The error message was: Attempt to invoke method: log() on undefined variable or class name: session : at Line: 2.
PS - Thanks for being totally quick on the replies. Hard to find that kind of service anymore.
Script not found error on headless machine
Hi,
The scripts are supposed to disappear from the import folder. That would mean that they were successfully imported.
Also, remember that the parameter that you pass with the -s switch should be the name of the script in screen-scraper, not the name of the XML file. That is, you should have a script in screen-scraper with the name "rev_lookup_scrape".
Feel free to post back if that still doesn't do it.
Kind regards,
Todd
Script not found error on headless machine
I tried that earlier as well, and not only does it not work that way, either, every time I run it, I get the same error and another problem...the scripts disappear from the import directory. Here's a fresh try.
old rev_lookup_scrape (Scraping Session).xml rev_lookup_script (Script).xml
root@foo:/usr/local/screen-scraper/import# cd ..
root@foo:/usr/local/screen-scraper# /usr/local/screen-scraper/jre/bin/java -jar screen-scraper.jar -s "rev_lookup_scrape"
Initializing screen-scraper...
The script "rev_lookup_scrape" was not found. Please check the name and try again.
root@foo:/usr/local/screen-scraper# ls import/
old
root@foo:/usr/local/screen-scraper# /usr/local/screen-scraper/jre/bin/java -jar screen-scraper.jar -s "rev_lookup_scrape"
Initializing screen-scraper...
The script "rev_lookup_scrape" was not found. Please check the name and try again.
I'm sure I must be missing something simple.
Script not found error on headless machine
Hi,
It appears as though your files simply aren't being imported into screen-scraper. This may be because you've modified the file names. Try exporting them from screen scraper, leaving the file names in tact (e.g., "rev_lookup_scrape (Script).xml"). When you start up screen-scraper from the command line, it should automatically import anything in that "import" folder. If it doesn't, it could be a permissions issue, or, in this case, possibly the file name.
As to documentation on Linux commands, there are really only a few things you'd do from the command line run screen-scraper in command line mode (which you obviously know how to do), and start/stop/restart the screen-scraper server. Could you give any other detail on where the documentation seems to be lacking? We'd be happy to fill in the holes.
Thanks much,
Todd
Script not found error on headless machine
Granted, I'm only a few hours into it, but I'm losing my mind. What else can I check?
Initializing screen-scraper...
The script "rev_lookup_scrape" was not found. Please check the name and try again.
Initializing screen-scraper...
The script "rev_lookup_script" was not found. Please check the name and try again.
rev_lookup_scrape.xml rev_lookup_script.xml rev_noscript.xml
cachedir doc import js.jar license.txt misc screen-scraper server start_server.sh workbench_lock
cobra.jar dom4j.jar INSTALL lax.jar locks resource screen-scraper.jar server_lock stop_server.sh wrapper
command_line_lock dom.jar jre lib log restart_server.sh screen-scraper.lax ss_pro_lin.tgz updater.jar
#This file is manipulated by screen-scraper. Edit it manually at your own risk!
#Tue Feb 06 15:49:11 PST 2007
ProxyForceAllHTTPRequestsToHTTPS=false
ExternalProxyPassword=
ExternalProxyUsername=
MailServerPassword=
Edition=PROFESSIONAL
MailServerUsername=
IPAddressesToAllow=192.168,127.0,localhost
MaxConcurrentScrapingSessions=5
TidyHTML=true
MailServerHost=
CommandLine.NumTimesRun=15
ConnectionTimeout=15
InstallDirectory=/usr/local/screen-scraper/
Workbench.NumTimesRun=0
ExternalProxyHost=
SaveLargeFields=false
OutputLogFiles=true
DataExtractorTimeout=1
MaximumMemoryAllocation=256
ExternalNTProxyHost=
DatabasePort=9001
MainFrame.LastWidth=800
ProxyPort=8777
Server.NumTimesRun=1
ExternalNTProxyPassword=
MainFrame.LastHeight=600
ExternalNTProxyUsername=
DontLogBinaryFiles=true
ExternalNTProxyDomain=
Version=3.0
DefaultProxySession=
ExternalProxyAuthentication=foo\:bar
ExternalNTProxyAuthentication=NTfoo\:NTbar
SOAPPort=8779
ExternalProxyPort=
LastSelectedDirectory=
ServerPort=8778
What am I missing? Also, is there a clear set of linux command line instructions? Those that I found on the site weren't as detailed as I had hoped.
Script not found error on headless machine
Hi Todd -
Thanks for the feedback and for the heads-up about Gentoo. I had some previous problems when using the default Blackdown jre, but I am using Sun's JDK 1.5 now. I am not sure what JRE is bundled with screen-saver, though.
It looks like the problem in my case was using the main directory (where the screen-scraper.jar is) instead of the import subdirectory for the scripts. Once I put the scripts in the import subdirectory, then at least for my very simple example it worked. It gives me something to build on. :)
Script not found error on headless machine
Hi,
Here are a couple of tips that will hopefully solve the problem
- Once you export a scraping session or script, don't rename it before dropping it into the "import" folder of another instance of screen-scraper.
- On your headless Linux box, you'll also need the "Hello World--begin scraping session" script. That is, you need to import the "Hello World" scraping session as well as the "Hello World--begin scraping session" script. The script invokes the scraping session, so they're both needed.
As to running on Gentoo, you're probably fine doing that as long as your scrapes aren't too long. We had a client not too long ago who had everything running on Gentoo, and ran up against some memory leak issues with the JVM. It's probably worth a shot, but you may want to consider that caveat.
Kind regards,
Todd Wilson