Can't get the search data to appear

Hi
I am trying to scrape www.euroauctions.com but when I inspect the site in Chrome I can find the URL that provides the data for the search results, but cannot seem to get it to run on SS.

I keep getting 500 server errors.
This is the URL and it has 3 parameters: filters, page & node ID (although there appears to be two records in the filter parameter (auctionId and masterCategoryId)

I am hoping there is a straightforward solution to this problem as it looks (relatively) straightforward thereafter?

Thanks for any help you can let me have in advance.

Jason

I see the site is using

I see the site is using ReCaptcha to arrest bots. It might be something you can work with, but it's probably going to be pretty complex, and require another service for captcha resolution.

Think I'll try harder to get an xml feed

Hopefully we won't come across too many of these!

Thanks

Jason

I have now got a json feed - stumped

The feed is one long list of 1500+ records with 20+ fields per record. I have written a pattern to extract the data which works, but if I use all of the tokens then it times out (too much going on, I assume.)

Is there a better way to step through the records and save as a csv file as I need to manipulate the data prior to using it?

Or is there a better JSON to csv converter?

Thanks
Jason

We ship the org.json library

We ship the org.json library with screen-scraper, and I use it when I need to read JSON.

Quick tutorial here: https://www.tutorialspoint.com/org_json/org_json_quick_guide.htm

Here is a sample where I make a new DataRecord, and have to parse each JSON node into the DataRecord.

import org.json.*;

DataRecord d    = session.getv("_DATARECORD");

JSONObject obj  = new JSONObject(dataRecord.get("JSON"));
JSONArray res   = obj.getJSONArray("result");
int hits        = res.getInt(0);
String con      = res.getString(1);

if (hits>0)
{
        JSONArray sobj  = new JSONArray(con);
        for (int i=0; i<sobj.length(); i++)
        {
                DataRecord dr           = new DataRecord();
                JSONObject party        = sobj.getJSONObject(i);
                dr.put("ROLE", party.getString("PARTYTYPE"));
       
                String filing           = StringUtils.substringBefore(party.getString("FILING"), " ");
                d.put("FILING_DATE", filing);
                filing = StringUtils.lowerCase(filing);
                filing = StringUtils.capitalize(filing);
                LocalDate filingDate = LocalDate.parse(filing, datePattern);
                d.put("FILING_DATE_CLEAN", filingDate);
       
                String name                     = party.getString("PARTY");
                name                            = sutil.stripHTML(name);
                name                            = StringUtils.normalizeSpace(name);
                if(!StringUtils.startsWithIgnoreCase(name, "Does "))
                {
                        log.log(">>>Parsing name: " + name);
                        String address  = "";
                       
                        String[] parts  = name.split("\\s\\d", 2);
                        log.logObjectByType(parts);
                       
                        name    = parts[0];
                        if (parts.length>1)
                        {
                                address = parts[1];
                        }
                       
                        dr.put("NAME", name);
                       
                        log.log("address: " + address);
                       
                        if (!sutil.isNullOrEmptyString(address))
                        {
                                dr.put("ADDRESS_RAW", address);
                                courtRecordsUtility.addDataForTable("locations", dr, true);
                        }
                        courtRecordsUtility.addDataForTable("people", dr, true);
                       
                        String attr     = party.getString("ATTORNEY");
                        attr            = sutil.stripHTML(attr);
                        attr            = StringUtils.normalizeSpace(attr);
                        if (!sutil.isNullOrEmptyString(attr) && !StringUtils.equalsIgnoreCase(attr, "Pro Per"))
                        {
                                DataRecord dr2  = new DataRecord();
                                dr2.put("ROLE", "Attorney");
                                dr2.put("NAME", attr);
                                dr2.put("PERSON_UID",caseUid + attr);
                                courtRecordsUtility.addDataForTable("people", dr2, true);
                        }
                }              
        }
        courtRecordsUtility.addDataForTable("cases", d, true);
        dm.flush();
}

Thank you for the info - might need a couple more pointers...

So I create a scrapeable file using the following URL (emailed for privacy)

The resulting response is very long - so what do I put in my '_DATARECORD' field - the whole thing or just the initial node as an example?

What is the input needed for the 'JSON' in your example here: JSONObject obj = new JSONObject(dataRecord.get("JSON"));

I think your example may be a little too high tech for me. All I am looking to do is take some of the fields from the node (images, make, model year, sale date etc.) Here is an example of one of the (1500+) nodes

{"auctionId":590,"ringAuctionId":11000590,"ringId":"D1R1","itemId":"140240862","lotNumber":"000000001010200001000140240862","controlNumber":"","sortOrder":"000000001010200001000140240862","masterCategoryId":"0004","masterCategory":"Commercial Vehicles","categoryId":"0005","category":"Commercials","title":"2013 Scania P400 ","titleHtml":null,"description":"2013 Scania P400 8x4 Tipper Lorry, Thompsons Body, Easy Sheet, Reverse Camera, A/C - EU63 FCG - YS2P8X40005330118","descriptionHtml":null,"startDateTime":null,"endDateTime":null,"location":"Leeds","processImages":false,"photos":[{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862.JPG?v=41956346","medium":null,"large":"/Piosys//Auction-0590/140240862.JPG?v=41956346","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_01.JPG?v=41956347","medium":null,"large":"/Piosys//Auction-0590/140240862_01.JPG?v=41956347","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_02.JPG?v=41956348","medium":null,"large":"/Piosys//Auction-0590/140240862_02.JPG?v=41956348","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_03.JPG?v=41956349","medium":null,"large":"/Piosys//Auction-0590/140240862_03.JPG?v=41956349","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_04.JPG?v=41956350","medium":null,"large":"/Piosys//Auction-0590/140240862_04.JPG?v=41956350","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_05.JPG?v=41956351","medium":null,"large":"/Piosys//Auction-0590/140240862_05.JPG?v=41956351","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_06.JPG?v=41956352","medium":null,"large":"/Piosys//Auction-0590/140240862_06.JPG?v=41956352","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_07.JPG?v=41956353","medium":null,"large":"/Piosys//Auction-0590/140240862_07.JPG?v=41956353","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_08.JPG?v=41956354","medium":null,"large":"/Piosys//Auction-0590/140240862_08.JPG?v=41956354","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_09.JPG?v=41956355","medium":null,"large":"/Piosys//Auction-0590/140240862_09.JPG?v=41956355","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_10.JPG?v=41956356","medium":null,"large":"/Piosys//Auction-0590/140240862_10.JPG?v=41956356","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_11.JPG?v=41956357","medium":null,"large":"/Piosys//Auction-0590/140240862_11.JPG?v=41956357","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_12.JPG?v=41956358","medium":null,"large":"/Piosys//Auction-0590/140240862_12.JPG?v=41956358","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_13.JPG?v=41956359","medium":null,"large":"/Piosys//Auction-0590/140240862_13.JPG?v=41956359","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_14.JPG?v=41956360","medium":null,"large":"/Piosys//Auction-0590/140240862_14.JPG?v=41956360","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_15.JPG?v=41956361","medium":null,"large":"/Piosys//Auction-0590/140240862_15.JPG?v=41956361","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_16.JPG?v=41956362","medium":null,"large":"/Piosys//Auction-0590/140240862_16.JPG?v=41956362","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_17.JPG?v=41956363","medium":null,"large":"/Piosys//Auction-0590/140240862_17.JPG?v=41956363","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_18.JPG?v=41956364","medium":null,"large":"/Piosys//Auction-0590/140240862_18.JPG?v=41956364","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_19.JPG?v=41956365","medium":null,"large":"/Piosys//Auction-0590/140240862_19.JPG?v=41956365","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_20.JPG?v=41956366","medium":null,"large":"/Piosys//Auction-0590/140240862_20.JPG?v=41956366","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_21.JPG?v=41956367","medium":null,"large":"/Piosys//Auction-0590/140240862_21.JPG?v=41956367","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_22.JPG?v=41956368","medium":null,"large":"/Piosys//Auction-0590/140240862_22.JPG?v=41956368","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_23.JPG?v=41956369","medium":null,"large":"/Piosys//Auction-0590/140240862_23.JPG?v=41956369","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_24.JPG?v=41956370","medium":null,"large":"/Piosys//Auction-0590/140240862_24.JPG?v=41956370","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_25.JPG?v=41956371","medium":null,"large":"/Piosys//Auction-0590/140240862_25.JPG?v=41956371","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_26.JPG?v=41956372","medium":null,"large":"/Piosys//Auction-0590/140240862_26.JPG?v=41956372","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_27.JPG?v=41956373","medium":null,"large":"/Piosys//Auction-0590/140240862_27.JPG?v=41956373","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_28.JPG?v=41956374","medium":null,"large":"/Piosys//Auction-0590/140240862_28.JPG?v=41956374","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_29.JPG?v=41956375","medium":null,"large":"/Piosys//Auction-0590/140240862_29.JPG?v=41956375","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_30.JPG?v=41956376","medium":null,"large":"/Piosys//Auction-0590/140240862_30.JPG?v=41956376","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_31.JPG?v=41956377","medium":null,"large":"/Piosys//Auction-0590/140240862_31.JPG?v=41956377","caption":null},{"thumbnail":"/Piosys//Auction-0590/Thumb/140240862_32.JPG?v=41956378","medium":null,"large":"/Piosys//Auction-0590/140240862_32.JPG?v=41956378","caption":null}],"mainImagesPath":null,"thumbImagesPath":null,"separator":null,"links":[],"REStreet":"Kellington","REStreet2":null,"RECity":"Leeds","REState":"","REZip":"DN14 0NY","RECounty":null,"RECountry":"GB","REAgent":null,"REAgentCompany":null,"REAgentPhone":null,"REAgentEmail":null,"REBedrooms":0,"REBathrooms":0.0,"RESquareFootage":0.0,"useGeocoding":false,"lat":0.0,"lng":0.0,"autoSerialNumber":"EU63 FCG - YS2P8X40005330118","autoIdNum":null,"autoYear":"2013","autoManufacturer":"Scania","autoModel":"P400","autoVIN":null,"autoOdometer":"449072 Kilometres ","autoOdometerUnits":null,"autoOdometer2":null,"autoOdometer2Units":null,"autoOdometer3":null,"autoOdometer3Units":null,"autoOdometer4":null,"autoOdometer4Units":null,"autoOdometer5":null,"autoOdometer5Units":null,"autoOdometer6":null,"autoOdometer6Units":null,"autoColor":null,"autoStatus":null,"autoNotes":null,"autoUserDefined1":"","autoUserDefined2":"","autoUserDefined3":"","autoUserDefined4":"","autoUserDefined5":"","autoUserDefined6":null,"autoSpecs":[{"section":"Measurements","spec":"Weight Kg","value":null},{"section":"Odometer Readings","spec":"Hours","value":null},{"section":"Odometer Readings","spec":"Km","value":"449072"},{"section":"Odometer Readings","spec":"Miles","value":null}],"cattleCountry":null,"cattleOversizeLoad":false,"cattleHeiferPriceBack":0.0,"cattleSupplementLot":false,"quantity":1,"sellerId":"100000089","sellerName":"Equipment Sales No.3 Ltd","reservePrice":0.0,"reserveMet":false,"disableReserveBidding":false,"auctionType":"One Lot","bidIncrement":0.0,"estimatesLow":0.0,"estimatesHigh":0.0,"startingBid":0.0,"silentDynamicExtension":0,"extended":false,"combinedId":0,"deleteBids":false,"receivedStatus":"On Site","newUnused":false,"fixedPrice":0.0,"hideImages":false,"featured":false,"status":null,"resetStatus":false,"currentBid":0.0,"notes":"","auctioneerNotes":"","auctioneer_notes":null,"hideSoldPrice":false,"acceptOffers":false,"disableAbsenteeBidding":false,"disableBuyItNow":false,"currencyCode":"GBP","inspectionReport":[]},

The start of the api shows: {"APIKey":"The key is in here","function":"GetItems","result":"OK","message":null,"data":[{"auctionId":590,"r.......

So I assume I am looking to use:

JSONArray array = obj.getJSONArray("data");

But I could just do with a couple more tips...

Many thanks

Jason

I don't know what more to

I don't know what more to point you to than that tutorial. You can sometimes make extractor patterns and pull things from JSON like that, but if that can't get you want you want, you need to parse it--the tutorial link I sent is the easiest one I've found.