Newbie Help with Parent/Child Relationships

My pleasure :)

Unfortunately, no, there isn't a handy editor for colors and indentation. I just cracked that up myself with HTML. The indentation can only be done by putting "non-breaking" spaces. To make a single one, you have to write "&nbsp;". I think I put 4 of those in a row in order to get a single indent, then 8 for two indents, etc, etc. Kind of hard to look at in plain old text while writing the reply :P The colors were a "<span style="color:blue>some text here</span>" , where the word "blue" can be replaced with lots of basic colornames, or with a 6-digit hex code, like "#000000" for black, or "#00ff00" for green.

Anyway.

Let me get one more thing clear..... When viewing an arbitrary page with a list of categories on it (a mix of end-of-the-line child categories and of new parent categories), the URLs are distinguished only by the parameters sent to the same page. So then, links on the page look like the following, for example:

Sean Connery, for $200 please.
fish heads, fish heads, rolly polly fish heads
boolean granola
breakfast cereals
Japan
10 little indians
squeeker
blah


You said that you've figured out how to do the initial scraping part, so I'll forgo further examination..

To keep copies of each category code as you go down the list might get a little tricksy. The best way that I've done this went something like this:

1. I'd start off with something in an initializing script that does this:
   
session.setVariable("DEPTH", 0);

   When to run:     Before scraping session begins    
2. Make sure you've got your extractor pattern(s) in place where you match for the category links. You'll be calling a couple of scripts from the extractor pattern.
   
/view.php?Screen=CTGY&~@DATARECORD@~">

   ~@DATARECORD@~: Don't save in session variable    Pattern: [^"]+
   Scripts:
       Process link        Squence: 1        After each pattern application    
       Redo category page (recursive)        Squence: 2        After each pattern application    
   Sub-extractor patterns:
   • 
Category_Code=~@CATEGORY_CODE@~

     ~@CHILD_CODE@~: Don't save in session variable    Pattern: [^&"]+
   • 
&Parent_Code=~@PARENT_CODE@~

     ~@PARENT_CODE@~: Don't save in session variable    Pattern: [^"]+

   This way, your pattern will match on every single link, whether it's for a child or parent category. The script that I've said to call ("Process link" from the main extractor pattern part) will have access to "CHILD_CODE" and "PARENT_CODE" via the dataRecord.

3. Now, for that script, "Process link". You'll have to keep track of each category code as a session variable with unique name, or else you'll be overwriting the category variable each time you find a new category code. So, that means you'll need a variable variablename. As in, make a call to "session.setVariable" and pass it a String variable for it's name, based on that "DEPTH" variable I said to create on step 1:
   Script: Process link in Interpreted Java
   
   String parentCode = dataRecord.get("PARENT_CODE");
   String childCode = dataRecord.get("CHILD_CODE");
   String depth = session.getVariable("DEPTH");

   // This will be the URL for the next category page you want to browse into
   session.setVariable("PARAMETERS", "Category_Code=" + childCode);
   session.setVariable("CHILD_AT_DEPTH_" + depth, childCode);

   if (parentCode != null)
   {
   session.setVariable("PARENT_AT_DEPTH_" + depth, parentCode);
   session.setVariable("PARAMETERS", session.getVariable("PARAMETERS") + "&Parent_Code=" + parentCode);
   }
   

   
   Now you've got a variable that contains only the parameters needed to get into the category. You don't want to include a parentCategory parameter if there isn't supposed to be one.
   Script: Redo category page (recurisve) in Interpreted Java
   
session.scrapeFile("The category-view scrapeableFile name here");

4. And then, your scrapeableFile for the category view should have a URL field kind of like the following:
   
http://www.someSite.com/view.php?Screen=CTGY&~#PARAMETERS#~


Now, I've left out something kind of critical... the "DEPTH" variable is never incremented. I'm not sure how you want to do that. If you increment it in the "Process list" script, you're going to find that every category on the site will have it's own depth number... so, it'd be more like an ID than a depth.

On the other hand, if you want to keep it as a true "depth" variable, where all child categories from the main page have the same depth number then you'd have to get even tricksier. It would involve adding 2 extra scripts:

1. One that goes "Before pattern application" on that main extractor pattern for the category scrapeableFile. It's "sequence" wouldn't matter, but to make it look right, I'd put it's sequence at "1", making the "Process link" script number 2.

   All this script would do is take the "DEPTH" variable and add 1 to it:
   
session.addToVariable("DEPTH", 1);


2. Another script would need to be run "After pattern is applied", which subtracts 1 from the DEPTH variable, and wipes out the old variables so that they don't carry over to any other categories by accident:
   
session.setVariable("CHILD_AT_DEPTH_" + session.getVariable("DEPTH"), null);
session.setVariable("PARENT_AT_DEPTH_" + session.getVariable("DEPTH"), null);
session.addToVariable("DEPTH", -1);


The recursive nature of the scrapeableFile would increment the DEPTH variable by 1 every time it begins a new nested category page. After it finishes, it'll decrement it by 1, putting it back where it needs to be for category list that called it. However, if a new nested category is encountered before the depth decrement, then the new recursive scrapeableFile will increment it *again* when it starts, and then decrement it back when it's finished. When you return to the previous scrapeableFile, it'll be okay and dandy.

With this method, you'll be overwriting the "PARENT_AT_DEPTH_X" and "CHILD_AT_DEPTH_X" (with "X" being the depth, not an "X") as you go down the list. So, the second category on the page will overwrite the first one's variables. But, so long as you're going down the child, sub-child, sub-sub-child tree, the variables will just be written as "..._DEPTH_1", ".._DEPTH_2", "..._DEPTH_3", etc, etc.

In other words, you'll have access to the whole line of category variables from depth 1 to depth X, so long as that's the line of variables that corresponds to the category branch that you're working down. And of course, the sessionVariable "DEPTH" is always available to you, to know how many categories you've nested yourself into.

Hope this is helping :)

Tim