screen-scraper support for licensed users

Could you recommend a workflow for scraping a site that requires 2FA using Microsoft Authenticator? Is this even possible?

EDIT: Apologies - ignore the question below - found the api URL using Chrome tools!

Hi - I have done hundreds of these scrapes now and can usually drill down and find where the data is hiding - but I have come across a site that won't release its secrets to me even with a proxy....

I assume there is a data layer hiding somewhere but I have no idea where?

Any pointers would be gratefully received.

Here is a list of them: https://www.cva-auctions.co.uk/auctions/

Here is the page I am interested in: https://www.cva-auctions.co.uk/stock?saleid=50

Many thanks

Jason

Hi

I am scraping an auction site that displays the sale date on the search results page, where I extract the detail page URL. On each detail page there is no mention of the sale date, so I cannot write it to the file in the standard way, (I am using the csv writer).

I can scrape the date from the search results page and save it as a session variable, I just have no idea how to write this variable against each record of the subsequent details page.

Results page with date:
http://stock.cva-auctions.co.uk/doncaster-truck-and-export-stock-list

Product page:

Until recently my SS Session worked. But now the Scrapable file won't find the extractor pattern. When I view the results in the SS Window, the dataset isn't empty but it IS missing the important information I need. When I open the Response in a web browser, i notice that it redirects to the data I need and in that source code the data is found.

Any idea why SS cannot see the same results as the web browser?

-----
The url I'm requesting is this:

I simply need to know how to extract part of the url that I am redirected to by the site, instead of the body of the website. Can you point me in the right direction please?

Hi I am getting the following error:

halting the scraping session because the maximum number of scripts allowed on the stack was reached. Current number on the stack is: 50.

Is there any way to increase the maximum number of scripts allowed?

Hi,

I am trying to connect to an url using the newly implemented http client curl.

From the logs i can see the following request is generated in screen-scraper:

curl "https://www.Electrolux.nl/"
--compressed
--insecure
--silent
--verbose
--get
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"
-H "Accept-Language: en-us,en;q=0.9"
-H "Expect:"

How do I make a json post request with screen-scraper? I need to load the following url (curl example)

curl "https://www.lg.com/nl/mkt/ajax/category/retrieveCategoryProductList" --data "categoryId=CT20182041&modelStatusCode=ACTIVE&bizType=B2C&viewAll=&filterFlag=Y&length=9&sort=&page=1&pagePosition="

Hi,

I downloaded the current version for Linux for screen-scraper Enterprise from:
https://www.screen-scraper.com/developers#download

but I can't seem to execute the shell executable - getting the error message:
"gzip: sfx_archive.tar.gz not in gzip format

I'am sorry, but the installer file seems to be corrupted.
If you downloaded that file please try again. If you transfer that file with ftp please make sure that you are using binary mode."

See the change log here: https://support.screen-scraper.com/alpha_log