Multithreading optimal sessions\threads mix
I have a setup that should allow multiple user simultaneous scrape. So, I was wondering what should I do?
Like install more instances running fewer threads, or install few instances running more threads? Can relate to the number of CPU cores?
Scrapes vary a lot in the
Scrapes vary a lot in the resources they need. Most of what you need to watch is the memory usage. Most of the time I will set an instance to run 10 threads and check memory usage, let it run for a while, and if memory still looks good add more threads. Once you see how many you can run in a single instance you can start evaluating parallel instances of screen-scraper.