Hi,

I am trying to pull all transactions from our Netsuite instance via API calls and ingest them into our datalake. I have tried using records, SuiteQL and now datasets but I am facing the same issues with volume of data.

There are over 400k records which is too much to call even with pagination (which can only pull 1000 records per call), and so the script will be running for hours – not appropriate for something we want to be as live a possible.

Is there a way to be able to pull data from a dataset but only take the last x days of data? I’ve tried filtering in the URL but no luck so far, there doesn’t seem to be any documentation on this specific use case.

eg the URL I am trying to call is something like

f’https://{ACCOUNT_ID}.suitetalk.api.netsuite.com/services/rest/query/v1/dataset/{DATASET_ID}/result?trandate AFTER “1/9/2024″‘

Any help would be greatly appreciated thanks!