Read from file (large amount of data)
Hello,
Please help me to understand how should I set scraper.
I have a .txt file with around 500000 urls I need to scrape.
I think about the following - I make one scrape and it's job is to read in the .txt input and loop, and then launch a separate scrape that will go get the data for each line. All this using RunnableScrapingSession
Is it good solution for such big amount? or you can suggest me something better
Really appreciate your help.
You should look into the
You should look into the samples we have. There is some buffered readers in there to handle large files.