Connection timeout error when validating proxies - "The host did not accept the connection within timeout of 9000 ms"
We are running some proxy gathering scrapers in server mode. We usually see a list of 2k or more to sort through.
During the validation phase I see the first 200 or so come back just fine and then I start getting the rest failing with a connection timeout error.
"The host did not accept the connection within timeout of 9000 ms"
In the script we have "proxyServerPool.filter( 9 );"
So that explains the 9000 ms part.
But why are the first 200 fine and then the rest fail?
Do you think I'm hitting a CentOS 5 limit on open connections or something in the Screen Scraper infrastructure?
Running in GUI mode and server mode in OS X seems to give the same results with 1200 of the 2000 passing.
Running in server mode on CentOS results in 188 of the 2000 passing.
Most of the proxies fail with error: "The host did not accept the connection within timeout of 9000 ms"
Increasing the filter to 30 "proxyServerPool.filter( 30 );" results in the error at about the same point (~200) out of 2000.
"The host did not accept the connection within timeout of 30000 ms"
So I don't think it is the filter timeout setting causing the issue.
Any thoughts?
Found the problem.
It turned out that the APF firewall was limiting the connections. Turning off the firewall removed the issue.
Now I just need to find the parameter that will allow more than 220 outgoing connections from a process.
Thanks for the other link on setting the file limits. I do see this periodically.
Part of the issue was needing to turn off IPV6, but the too many open files was still periodically appearing.
This should help that situation.
Thanks!
This FAQ might also
This FAQ might also help:
http://community.screen-scraper.com/node/1078
Also, there's no set limit within screen-scraper as to how many connections and such it can handle. It's really limited by the underlying operating system and hardware.
If you check the files in
If you check the files in your screen-scraper/log directory, there should be something therein to give us a direction. Look specifically for "too many files open" or something in that vein.