ProxyServerPool

Overview

The proxy server pool object is used to aid with manual anonymization of scrapes. An example of how to setup manual proxy pools is available in the documentation. You will likely want to read that page first if you are new to the process.

Additionally, you should reference the available method's available in the Anonymous API

ProxyServerPool

ProxyServerPool ProxyServerPool ( )

Description

Initiate a ProxyServerPool object.

Parameters

This method does not receive any parameters.

Return Values

Returns a ProxyServerPool.

Change Log

Version Description
4.5 Available for all editions.

Class Location

com.screenscraper.util.ProxyServerPool

Examples

Creating ProxyServerPool

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object. This object will
 // control how screen-scraper interacts with proxy servers.

 proxyServerPool = new ProxyServerPool();

filter

void proxyServerPool.filter ( int timeout )

Description

Set the timeout that will render a proxy as being bad.

Parameters

  • timeout Number of seconds before timeout, as an integer.

Return Values

Returns void.

Change Log

Version Description
4.5 Available for all editions.

Examples

Setup Timout for Bad Proxies

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object.
 proxyServerPool = new ProxyServerPool();

 // Must be set on the session before other calls are made
 session.setProxyServerPool(proxyServerPool);

 // This tells the pool to populate itself from a file
 proxyServerPool.populateFromFile( "proxies.txt" );

 // Validate proxies up to 25 proxies at a time.
 proxyServerPool.setNumProxiesToValidateConcurrently( 25 );

 // This method call tells screen-scraper to filter the list of>
 // proxy servers using 7 seconds as a timeout value. That is,
 // if a server doesnt respond within 7 seconds, it's deemed
 // to be invalid.

 proxyServerPool.filter( 7 );

getNumProxyServers

int proxyServerPool.getNumProxyServers ( int numProxyServers )

Description

Retrieve the number of available proxy servers.

Parameters

This method does not receive any parameters.

Return Values

Returns the number of available proxy servers, as an integer.

Change Log

Version Description
4.5 Available for all editions.

Examples

Write Good Proxies to File

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object.
 proxyServerPool = new ProxyServerPool();

 // Must be set on the session before other calls are made
 session.setProxyServerPool(proxyServerPool);

 // This tells the pool to populate itself from a file
 proxyServerPool.populateFromFile( "proxies.txt" );

 // Validate proxies up to 25 proxies at a time.
 proxyServerPool.setNumProxiesToValidateConcurrently( 25 );

 // Set timout interval
 proxyServerPool.filter( 7 );
 
 // Check number of available proxies
 if (proxyServerPool.getNumProxyServers() < 4) {
 {
    // Repopulate available proxies
    proxyServerPool.setRepopulateThreshold( 5 );
 }

outputProxyServersToLog

void proxyServerPool.outputProxyServersToLog ( )

Description

Write list of proxies to log.

Parameters

This method does not receive any parameters.

Return Values

Returns void.

Change Log

Version Description
4.5 Available for all editions.

Examples

Write Good Proxies to File

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object.
 proxyServerPool = new ProxyServerPool();

 // Must be set on the session before other calls are made
 session.setProxyServerPool(proxyServerPool);

 // This tells the pool to populate itself from a file>
 proxyServerPool.populateFromFile( "proxies.txt" );

 // Validate proxies up to 25 proxies at a time.
 proxyServerPool.setNumProxiesToValidateConcurrently( 25 );

 // Set timout interval
 proxyServerPool.filter( 7 );

 // Write good proxies to file
 proxyServerPool.writeProxyPoolToFile( "good_proxies.txt" );

 // You might also want to write out the list of proxy servers
 // to screen-scraper's log.

 proxyServerPool.outputProxyServersToLog();

populateFromFile

void proxyServerPool.populateFromFile ( String filePath )

Description

Add proxy servers to pool using a text file.

Parameters

  • filePath Path to the file containing proxy settings, as a string. The format of the file is a hard return delimited list of domain:port listing.

Return Values

Returns void.

Change Log

Version Description
4.5 Available for all editions.

Examples

Creating ProxyServerPool

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object. This object will
 // control how screen-scraper interacts with proxy servers.

 proxyServerPool = new ProxyServerPool();

 // Must be set on the session before other calls are made
 session.setProxyServerPool(proxyServerPool);

 // This tells the pool to populate itself from a file
 // containing a list of proxy servers. The format is very
 // simple--you should have a proxy server on each line of
 // the file, with the host separated from the port by a colon.
 // For example:
 // one.proxy.com:8888
 // two.proxy.com:3128
 // 29.283.928.10:8080
 // But obviously without the slashes at the beginning.

 proxyServerPool.populateFromFile( "proxies.txt" );

setAutomaticProxyCycling

void setAutomaticProxyCycling ( boolean cycleProxies )(professional and enterprise editions only)

Description

Enables or disables automatic proxy cycling. When this is set to false (default is true) the current proxy that was automatically selected from the pool will be used each time the next proxy is requested. When set to true, each call to the getNextProxy method will cycle as normal between all available proxies.

Parameters

A boolean value.

Return Value

None

Change Log

Version Description
5.5.17a Available in Professional and Enterprise editions.

Example

// Assuming a ProxyServerPool object was created previously, and
// stored in the PROXY_SERVER_POOL session variable.
pool = session.getv( "PROXY_SERVER_POOL" );

// This will cause the current proxy server to be reused until the
// value is set back to true.
pool.setAutomaticProxyCycling( false );

// The corresponding getter will indicate what the current value is.
session.log( "Automatically cycling proxies: " + pool.getAutomaticProxyCycling() );

setNumProxiesToValidateConcurrently

void proxyServerPool.setNumProxiesToValidateConcurrently ( int numProxies )

Description

Set the number of proxies that can be tested concurrently.

Parameters

  • numProxies Number of proxies to be validated concurrently, as an integer.

Return Values

Returns void.

Change Log

Version Description
4.5 Available for all editions.

Examples

Test Proxies in Pool in Multiple Threads

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object.
 proxyServerPool = new ProxyServerPool();

 // Must be set on the session before other calls are made
 session.setProxyServerPool(proxyServerPool);

 // This tells the pool to populate itself from a file
 proxyServerPool.populateFromFile( "proxies.txt" );

 // screen-scraper can iterate through all of the proxies to
 // ensure theyre responsive. This can be a time-consuming
 // process unless it's done in a multi-threaded fashion.
 // This method call tells screen-scraper to validate up to
 // 25 proxies at a time.

 proxyServerPool.setNumProxiesToValidateConcurrently( 25 );

setRepopulateThreshold

void proxyServerPool.setRepopulateThreshold ( int repopulateThreshold )

Description

Set threshold to get more proxy servers.

Parameters

  • repopulateThreshold Lowest number of proxies before more proxies are requested.

Return Values

Returns void.

Change Log

Version Description
4.5 Available for all editions.

Examples

Write Good Proxies to File

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object.
 proxyServerPool = new ProxyServerPool();

 // Must be set on the session before other calls are made
 session.setProxyServerPool(proxyServerPool);

 // This tells the pool to populate itself from a file
 proxyServerPool.populateFromFile( "proxies.txt" );

 // Validate proxies up to 25 proxies at a time.
 proxyServerPool.setNumProxiesToValidateConcurrently( 25 );

 // Set timout interval
 proxyServerPool.filter( 7 );

 // Write good proxies to file
 proxyServerPool.writeProxyPoolToFile( "good_proxies.txt" );

 // Write Proxy Servers to log
 proxyServerPool.outputProxyServersToLog();

 // As a scraping session runs, screen-scraper will filter out
 // proxies that become non-responsive. If the number of proxies
 // gets down to a specified level, screen-scraper can repopulate
 // itself. Thats what this method call controls.

 proxyServerPool.setRepopulateThreshold( 5 );

writeProxyPoolToFile

void proxyServerPool.writeProxyPoolToFile ( String path )

Description

Write list of proxies after invalid proxies have been removed.

Parameters

  • path File path to where the file should be written, as a string.

Return Values

Returns void.

Change Log

Version Description
4.5 Available for all editions.

Examples

Write Good Proxies to File

 import com.screenscraper.util.*;

 // Create a new ProxyServerPool object.
 proxyServerPool = new ProxyServerPool();

 // Must be set on the session before other calls are made
 session.setProxyServerPool(proxyServerPool);

 // This tells the pool to populate itself from a file
 proxyServerPool.populateFromFile( "proxies.txt" );

 // Validate proxies up to 25 proxies at a time.
 proxyServerPool.setNumProxiesToValidateConcurrently( 25 );

 // Set timout interval
 proxyServerPool.filter( 7 );

 // Once filtering is done, it's often helpful to write the good
 // set of proxies out to a file. That way you may not have to
 // filter again the next time.

 proxyServerPool.writeProxyPoolToFile( "good_proxies.txt" );