Looping a script
I have two questions:
For some reason my script stopped writing data to the csv file after 1565 times. The log was still doing everything it was supposed to do. Do you know why that could be?
I wanted to write a script that would let me do it in batches of 1000 at a time. I found this code in another posting but it does not seem to do anything? What am I doing wrong?
for( int i = 1; i < 1000; i++ )
{
session.scrapeFile( "6 Get Parts Details" );
}
Any help is much appreciated. Thanks, rudy
Thanks for your help.
Thanks for your help. unfortunately I cannot use addToVariable in my version of screen-scraper.
Let me try to explain what I am trying to do. I have a set up much like the one in the tutorial. First it looks for the model number of a product in scrapeble #2. After each pattern match on the extractor page it runs script file #3. Script file #3 simply refers to scrapable file #4. Scrapable file #4 extracts the parts details for each model.
I am trying to set it up so that it will stop after the first 1000 models.
The way I set it up it appears that it does not keep the i value in its memory. I added in the line to write i to the session log but it seems that each time it moves to a new model it resets i. How do i get it to keep incremementing i throughout the entire scraping session?
Code:
for( int i = 1; i < 3; i++ )
{
session.log( "i equals " + i);
session.scrapeFile( "4 Get Parts" );
}
I think that your i might be out-of-scope
Hi Rudy,
I think I might see the problem: your counter i is out-of-scope, which is a fancy way of saying that any variables (like i) that you declare within a script exist only while that script is active, unless you write it to a session variable (which exist at the next-higher level of the scraping session vs. the script). As a result, each time the script is triggered, it starts a new instance of your counter i.
One way around this is to declare a variable for your counter in an initialization script that gets run before the scrape, such as:
Then in your script file #3, you would retrieve the variable (and likely convert it to a integer, as I think session variables are usually saved as strings), check to see if it's <1000, and then decide what to do; this might look something like:
if (Tool_Counter_INT <1000)
{
//if it's <1000, scrape the file and then increment the counter
session.scrapeFile(...) //this scrapes the file
Tool_Counter_INT = Tool_Counter_INT++; //this increments the counter by 1
session.setVariable("Tool_Counter_Value", String.valueOf(Tool_Counter_INT)); //this saves the updated counter to the session variable declared in the init script
} else {
//otherwise, stop the scrape (if this is what you want to do...)
session.stopScraping(); //this stops the scrape once it hits 1000 parts
session.logError("Scrape hit 1000 parts, stopping...");
}
Let me know if this makes any sense and/or answers your question (as my understanding could be wayyyyyyy off...)
Regards,
Justin
Another error message
I think this is really moving in the right direction. However, I am getting this error message:
ERROR--Paddles: An error occurred while processing the script: Paddles 3 Go To Model
Paddles: The error message was: class bsh.EvalError (line 5): Integer .parseInt ( session .getVariable ( "Tool_Counter_Value" ) ) -- Typed variable declaration : Error in method invocation: Static method parseInt( java.lang.Integer ) not found in class'java.lang.Integer'
I have the script in Interpreted Java.
That's a weird one...
Hi Rudy,
Hmm...the error message is saying that it can't find the function parseInt in the Java libraries it has available. I've never had to declare anything in java.lang library before as it's built into SS. There's no space between Integer & parseInt, right? (FYI, there shouldn't be)
Let's try something; can you add the following lines to the very top of your script:
import java.lang.Integer;
import java.lang.Integer.*;
These lines force the script to import the library containing the parseInt function. They need to be at the very top of the script so they can be imported beforehand. If you try this and it still doesn't work, try modifying the parseInt line as follows:
Can you try adding these and let us know what happens?
Regards,
Justin
Did not work
I tried both solutions and it did not work. I have copied the error message and the code. Any suggestions?
Error message:
Processing script: "Paddles 3 Go To Model"
ERROR--Paddles: An error occurred while processing the script: Paddles 3 Go To Model
Paddles: The error message was: class bsh.EvalError (line 7): int Tool_Counter_INT = new Integer .parseInt ( session .getVariable ( "Tool_Counter_Value" ) ) -- Typed variable declaration : Class: Integer.parseInt not found in namespace
First 9 lines of code:
// This will scrape Details page
import java.lang.*;
import java.lang.Integer;
import java.lang.Integer.*;
int Tool_Counter_INT = new Integer.parseInt(session.getVariable("Tool_Counter_Value")); //this converts the counter variable to an integer
if (Tool_Counter_INT <3)
Looks like I was too clever for my own good...
Hi Rudy,
I tried putting in the code I supplied you in a scrape that I'm running and was able to replicate your error. It turns out that you don't need to convert the counter from a string to an int all the time; when you assign a number to a SS variable, it stays a number (not a string as I thought). Here's some code for your script #3 that might work better:
if (Tool_Counter_INT <1000)
{
//if the counter is <1000, scrape the file and then increment the counter by 1
session.scrapeFile(...) //this scrapes the file
Tool_Counter_INT = Tool_Counter_INT + 1; //this increments the counter by 1
session.setVariable("Tool_Counter_Value", Tool_Counter_INT); //this saves the updated counter to the session variable declared in the init script
} else {
//otherwise, stop the scrape (if this is what you want to do...)
session.stopScraping(); //this stops the scrape once it hits 1000 parts
session.logError("Scrape hit 1000 parts, so stopping...");
}
Can you try this and let us know what happens? FYI, you can remove the 3 import statements that I asked you to add; these aren't necessary anymore.
Thanks,
Justin
This worked!!!
I added a semi colon in line 7 and it worked (just in case other people will use this post in the future). Thank you so much. One more question.
Could I also use a list of the model numbers in the code in the script file as an array? Do you have an example of how I could do that?
Phew...
Glad to hear that this worked and thanks for correcting my mistake.
As for arrays, they're not hard to set up but I'm not sure I understand what you're going to do with it. Do you want to compare the part number on the page with a standard value?
FWIW, you set up a (string) array as follows:
NOTE: Make sure you use curly {} brackets vs. regular () or square [] brackets. The latter 2 won't work here.
Once the array is declared, use the following code to retrieve an element from it:
NOTE 1: Make sure you use square [] brackets to call the array element.
NOTE 2: Arrays are 0-based meaning that you'd enter 0 as the element number to retrieve the first element.
NOTE 3: Once you declare an array, you can't change or modify it while the scrape is running. If you need an array that you can change on the fly, look into using an "Arraylist" rather than an array.
You can also loop through the elements in an array; this is via a pretty simple for-loop w/the above code, so I won't bother writing it out here.
HTH!
I don't see a particular
I don't see a particular problem in that loop. It might be due to where the script is invoked. For example, if you have the script running somewhere on scrapeableFile "6 Get Parts Details", this script executes each time. If that were the case, you might
{
for (i=0; i<1000; i++)
{
session.addToVariable("PAGE", 1);
session.scrapeFile("6 Get Parts Details");
}
}