Home » Forums » screen-scraper » screen-scraper public support

replace error

I use the replace function:
session.setVariable("WIND",session.getVariable("WIND").replaceAll(",","."));
session.setVariable("city",session.getVariable("city").replaceAll(",","."));
session.setVariable("max",session.getVariable("max").replaceAll(",","."));
session.setVariable("min",session.getVariable("min").replaceAll(",","."));
session.setVariable("temp",session.getVariable("temp").replaceAll(",","."));

but sometime wind variables are empty (the data in the web page is missing) and the program provide me this error:

An error occurred while processing the script: save data
The error message was: class bsh.EvalError (line 41): .replaceAll ( "," , "." ) -- Attempt to invoke method replaceAll on null value

How can avoid this problem?
This error stops the saving data in a csv file and I loss all the scraped city data
thx for your support
Fabio

‹ Looping a script Access to java jar files ›
fabio on 09/08/2010 at 3:59 pm

I'm having a hard time with

I'm having a hard time with this function. I'm trying to remove all of the words "logo" from some file sources. For example: C::/Program files/Screen-Scraper/Logo-340x83t.jpg

Here is my code:

DESCRIPTION=session.getv("DESCRIPTION");
DESCRIPTION = DESCRIPTION.replace("logo,", " ");

It doesn't seem to pull anything out. Possibly because of the leading "/" ?

I've read around on some java script forums and I've seen to add a /g but that returns an error for me.

Thanks
Ben

 allthenamesweretaken on 12/04/2012 at 3:25 pm

I think you're just missing

I think you're just missing an "All", because this works

testStr = "There is a user logo";
testStr = testStr.replaceAll("logo", " ");
session.log(testStr);

 jason on 12/04/2012 at 5:05 pm

I'm really sorry I still

I'm really sorry I still can't get it to work. I've tried both replace and replaceAll

To be fair my string is extremely long as-in a page of HTML code. I'm trying to extract certain words from file sources.

Here is an instance of "logo" that I am still finding.

='sh-logo u-flL

I have a lot of other words that i'm trying to remove and having the same results.

Also this particular string is never a null so I don't need to add an exception but when I do need one I always us the sutil.nullToEmptyStrin

Thanks Again,
Ben

 allthenamesweretaken on 12/05/2012 at 2:14 pm

Figured it out. I had an

Figured it out. I had an extra comma in there.
DESCRIPTION = DESCRIPTION.replaceAll("Logo,", "");

updated to DESCRIPTION = DESCRIPTION.replaceAll("Logo", "");

thanks

 allthenamesweretaken on 12/06/2012 at 12:05 pm

You can't use replaceAll on a

You can't use replaceAll on a null, so the thing to do is check that the value isn't null first. I use a function like this a lot:

// Fix format issues.
String fixString(String value)
{
        if (value != null)
        {
                value = value.replaceAll("\"", "\'");
                value = value.replaceAll("&", "&");
                value = value.replaceAll(" ", " ");
                value = value.replaceAll("\\s{2,}", " ");
                value = value.trim();
        }
        return value==null? "" : value;
}

session.setVariable("TEST", fixString(session.getVariable("TEST")));
session.setVariable("EXAMPLE", fixString(session.getVariable("EXAMPLE")));

 jason on 09/09/2010 at 11:22 am

Challenge of the day

Hey Justin I thought I could give you a challenge. I'm trying to do this same thing but trim words rather than charachters.

For example "Big Beautiful Pink Purse with lots of Pink"

The string has 8 words. Can I trim this to 4 words?

What I am doing is a search but the 8 word example is two specific so I would like to trim it down with out ending up with something like:"Big Beautiful Pink Purse wi" Which doesn't return any results.

I found this on a on the internet:

a=dataRecord.get("TITLE");

function trim(string, length) {
var trimmed = string.split(' ').slice(0, length),
idx = 0;

for (string = ""; idx < trimmed.length; string += trimmed[idx++] + ' ');

return string;
}
session.log( trim(a, 5) );

I tried to udjust it to my specs ut I don't really understand it. Im not really a code master,

 allthenamesweretaken on 02/14/2013 at 5:03 pm

Ahh, you know that I love a challenge...

I see what you're trying to do but I'm not sure that you are going about it in the right way. It would be pretty easy to code a solution that just returned the beginning 4 or less words from a phrase, but that's not what you want to do: you want SS to extract the most useful part of a phrase (e.g., Pink Purse, Pink Leather Purse, Big Purse, etc.).

What's important to understand here is that the number of words in that phrase really doesn't matter - rather, it's getting rid of the non-important/non-relevant words in the phrase (e.g., Big, with lots of, etc.) before you send it to a search engine. The way I would do it is to first look at a number of titles to figure out what words you could remove and then code a solution that strips these out of the search string that's then submitted to Google/Yahoo/etc. The actual coding would be similar to the replaceAll solution above, but you would use your list of non-relevant words as replacement strings.

If you find that the number of non-relevant words is too big, another alternative strategy would be to look at words that identify the break between the main, relevant part of the title (e.g., Big Pink Purse) and the non-relevant part(s) (e.g., with lots of Pink); in this example it's 'with' that's the break-point. You could then break the string into two parts and submit both to Google or Yahoo to see which brings back the more relevant results.

Sorry I can't provide you with some magic code (I'm under a deadline (actually a few deadlines)), but it would probably help you to develop a search strategy first and then add code to your scrape to implement this strategy rather than 'code first and hope for the best'.

Good luck and let me know if you have any questions...

Regards,
Justin

 Justin_S on 02/15/2013 at 7:23 am

Supereasy. We have a library

Supereasy. We have a library for Apache stringUtils and wordUtils included now.

 jason on 02/15/2013 at 4:47 pm

Really? That's great!

Hi Jason,

It's good to know you have these utils included - can you point us to the documentation so we can see how to use it?

Regards,
Justin

 Justin_S on 02/26/2013 at 11:09 am

Just see the stringUtils

Just see the stringUtils docs:

http://commons.apache.org/proper/commons-lang//apidocs/org/apache/commons/lang3/StringUtils.html

 jason on 02/27/2013 at 11:05 am

Don't forget sutil.nullToEmptyString()!

I tend to use the provided sutil.nullToEmptyString method to do null tests/conversion. For example...

session.setVariable("WIND",session.getVariable("WIND").replaceAll(",","."));

...could be updated to...
session.setVariable("WIND",(sutil.nullToEmptyString(session.getVariable("WIND")).replaceAll(",","."));

...which would replace a null with an empty string before the replaceAll method is applied. FWIW, I do appreciate the sutil methods as they can be big timesavers for certain operations - I just wish I could afford the Pro/Enterprise version so I cuse use them all. :(

 Justin_S on 12/05/2012 at 8:17 am

Hey Justin, did you see the

Hey Justin, did you see the end-of-year sale? It's as good a sale as I ever remember having ...

 jason on 12/05/2012 at 10:12 am

Wish I could...

I did see the sale, but the budget is controlled by the (cheap-skate) law firm I work for... :-(

 Justin_S on 12/07/2012 at 12:30 pm

Understood, but since you're

Understood, but since you're a good bloke and always helping people out on my forum, let me know if there is something I can do to put a license in your reach ...

 jason on 12/07/2012 at 2:55 pm