Home » Forums » screen-scraper » screen-scraper support for licensed users

Strange license error when using scrapeableFile.setUserAgent

I am trying to change the browser user agent so I have this script which I call "Before file is scraped":

Random randomGenerator = new Random();

int randomnumber = randomGenerator.nextInt(32);

switch(randomnumber){
        case 0 : scrapeableFile.setUserAgent( "Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en]" );
        case 1 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1" );
        case 2 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Win98; en-US; rv:x.xx) Gecko/20030423 Firebird Browser/0.6" );
        case 3 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Win9x; en; Stable) Gecko/20020911 Beonex/0.8.1-stable" );
        case 4 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/0.2.153.1 Safari/525.19" );
        case 5 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060731 Firefox/1.5.0.5 Flock/0.7.4.1" );
        case 6 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008092215 Firefox/3.0.1 Orca/1.1 beta 3" );
        case 7 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x" );
        case 8 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xx) Gecko/20030504 Mozilla Firebird/0.6" );
        case 9 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.xxx) Gecko/20041027 Mnenhy/0.6.0.104" );
        case 10 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5" );
        case 11 : scrapeableFile.setUserAgent( "Mozilla/5.0 (Windows; U;XMPP Tiscali Communicator v.10.0.1; Windows NT 5.1; it; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" );
        case 12 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; Linux i686; U;rv: 1.7.13) Gecko/20070322 Kazehakase/0.4.4.1" );
        case 13 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux 2.4.2-2 i586; en-US; m18) Gecko/20010131 Netscape6/6.01" );
        case 14 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.8.0.2) Gecko/20060309 SeaMonkey/1.0" );
        case 15 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.7.6) Gecko/20050405 Epiphany/1.6.1 (Ubuntu) (Ubuntu package 1.0.2)" );
        case 16 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-US; Nautilus/1.0Final) Gecko/20020408" );
        case 17 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801" );
        case 18 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021007 Phoenix/0.3" );
        case 19 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040413 Epiphany/1.2.1" );
        case 20 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot" );
        case 21 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061129 BonEcho/2.0" );
        case 22 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/20061205 Iceweasel/2.0.0.1 (Debian-2.0.0.1+dfsg-2)" );
        case 23 : scrapeableFile.setUserAgent( "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9a8) Gecko/2007100619 GranParadiso/3.0a8" );
        case 24 : scrapeableFile.setUserAgent( "Mozilla/5.0 Galeon/1.0.2 (X11; Linux i686; U;) Gecko/20011224" );
        case 25 : scrapeableFile.setUserAgent( "Opera/6.x (Linux 2.4.8-26mdk i686; U) [en]" );
        case 26 : scrapeableFile.setUserAgent( "Opera/6.x (Windows NT 4.0; U) [de]" );
        case 27 : scrapeableFile.setUserAgent( "Opera/7.x (Windows NT 5.1; U) [en]" );
        case 28 : scrapeableFile.setUserAgent( "Opera/8.xx (Windows NT 5.1; U; en)" );
        case 29 : scrapeableFile.setUserAgent( "Opera/9.0 (Windows NT 5.1; U; en)" );
        case 30 : scrapeableFile.setUserAgent( "Opera/9.00 (Windows NT 5.1; U; de)" );
        case 31 : scrapeableFile.setUserAgent( "Opera/9.60 (Windows NT 5.1; U; de) Presto/2.1.1" );
}

session.logError("Changed browser user agent");

However I'm getting this error when running the script:

ERROR--MyScrapingSession: An error occurred while processing the script: common - randomize browser user agent
MyScrapingSession: The error message was: InvalidVersionException (line 33): MyScrapingSession: The "removeHTTPHeader(String)" method is not available in this edition of screen-scraper. It requires at least the "Enterprise" edition.--Sourced file: inline evaluation of: ``Random randomGenerator = new Random(); int randomnumber = randomGenerator.nextI . . . '' : Method Invocation scrapeableFile.setUserAgent

Why am I getting this error? I have the professional license and I should not have a problem using setUserAgent, right?

Cheers,
Boga

‹ "Your connection is not private" when visiting Google through SS workbench Proxy java.lang.RunTimeException Could not generate DH Keypair ›
bogavante on 10/13/2015 at 12:42 am

The error says you're calling

The error says you're calling scrapeableFile.removeHTTPHeader(), but I don't see it in your script.

However ... there is an easier way in the alpha documentation.

Make a CSV with a row for each user agent you want to pick from. Name it user_agents.csv and put in the screen-scraper/input directory. Then just

userAgent = sutil.getRandomUserAgent();
session.log("==>Using agent " + userAgent);
scrapeableFile.setUserAgent(userAgent);

 jason on 10/13/2015 at 9:48 am

I did exactly as you

I did exactly as you suggested, pasting your code on a new script and put this new one instead of the old one. Also put the user agents list in the location you said.

I get the same error complaining about removeHTTPHeader(String) requiring the Enterprise license. Error pops up on line 3, which is the call to scrapeableFile.setUserAgent(userAgent);

 bogavante on 10/13/2015 at 11:11 am

the only change I did in your

the only change I did in your code was I changed the session.log to session.logError. Here is the error:

==>Using agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
ERROR--MyScrapingSession: An error occurred while processing the script: randomize user agent
MyScrapingSession: The error message was: InvalidVersionException (line 3): MyScrapingSession: The "removeHTTPHeader(String)" method is not available in this edition of screen-scraper. It requires at least the "Enterprise" edition.--Sourced file: inline evaluation of: ``userAgent = sutil.getRandomUserAgent(); session.logError("==>Using agent " + use . . . '' : Method Invocation scrapeableFile.setUserAgent

 bogavante on 10/13/2015 at 11:15 am

Just in case it wasn't clear,

Just in case it wasn't clear, eventhough the error says otherwise, I not calling removeHTTPHeader anywhere in my scrapes. In fact I never heard of it.

 bogavante on 10/14/2015 at 8:07 am

It looks like this is a bug.

It looks like this is a bug. We'll look for a fix and will need to release it in a new alpha version.

 jason on 10/14/2015 at 10:13 am

Isn't it possible that you

Would it be possible that you issue me an Enterprise license for temporary use until this gets fixed?

I'm at a dead-end scraping a very strict website and I really want to try changing my user agent to see if that helps before taking another route.

I won't use any of the enterprise features, just this buggy function that should work with my professional license.

Thanks,
Boga

 bogavante on 10/14/2015 at 10:30 am

I just attached a new

I just posted a new screen-scraper.jar for you. You need to:

  1. Make sure screen-scraper is not running
  2. Download the attached screen-scraper.jar
  3. Overwrite the screen-scraper.jar in your install
  4. Launch screen-scraper and try again.
 jason on 10/14/2015 at 10:40 am

great. The error disappeared

great. The error disappeared now with the new .jar

Any measure I must take in the future with my now modified installation or will I be able to update new versions normally?

Thanks,
Boga

 bogavante on 10/14/2015 at 12:22 pm

The next alpha will include

The next alpha will include this fix too, so you should be fine to upgrade as usual.

 jason on 10/15/2015 at 9:59 am