2: Record Search Results

Create Proxy Server

As in the first tutorial, we'll be recording a browser session using the proxy server. Remember that the proxy session records all of the HTTP requests and responses that take place between your browser and the target server.

Create a new proxy session now by clicking the New Proxy Session button and name it Shopping Site.

Configure Proxy Server

At this point start up the proxy server by clicking the Start Proxy Server button, then configure your web browser as you did in the first tutorial (if needed use our documentation on configuring your browser).

In your web browser (after configuring it to use the proxy server) go to this URL: http://www.screen-scraper.com/shop/.

Remember that you may want to use one browser with the proxy server and one to view the tutorial.

Make sure to check Don't record binary files.

Record Pages

Search Results Page

We'll be doing a search in the shopping web site for the term "dvd" in the product catalog. Do this by typing dvd into the search box located in the upper-right corner of the home page, then click the Search button. You'll see screen-scraper work for a bit, then, once it finishes, you should see one new row in the HTTP Transactions table.

Add a note, Home page to the appropriate transaction by double-clicking in the notes field and typing it.

Add a note, Search results to the appropriate transaction by double-clicking in the notes field and typing it.

Next Page

We'll want to traverse all of the search results, so, in your web browser, click the Next >> link.

Add a note, Next page.

Details Page

Later on we'll be scraping the details pages, so let's record one of those now. Click on the link for Speed to view the details on this DVD.

Add a note, Details page.

Stop Proxy Session

These are the only pages we're interested in at this point, so go ahead and stop the proxy session by clicking the Stop Proxy Server button on the General tab. You'll also want to re-configure your browser so that it's no longer using screen-scraper as a proxy server.

Note column in HTTP Transactions table.