external service - how to use $skiptoken (server side paging / 1000 recods)

[nicorunge]

Hi all,

I'm trying to call all Users from a SuccessFactors System. This is my current call:

        const sfsfSrv = await cds.connect.to('sfsf')
        const { User } = sfsfSrv.entities
        const q = SELECT.from(User)
        const data = await sfsfSrv.run(q)

But SF is returning max 1000 records in a single call (server side paging). At the end of 1000 records, it returns a __next property which contains a $skiptoken value, to fetch the next 1000 users.

"__next": "https://api.successfactors.com/odata/v2/User?$skiptoken=eyJzdGFydFJvdyI6MTAwMCwiZW5kUm93IjoyMDAwfQ=="

I've read in the sap-cloud-sdk docs, that it automatically resolves this server side paging: "By default, the SAP Cloud SDK automatically resolves all pages of a result-set if server-driven paging is encountered."

But this seems not work in CAP, although it is using the sap-cloud-sdk under the hood?!
Can I manually trigger a second call by providing the $skiptoken to my select statement? How would the syntax look?

I could do something like this, but it will return data in JSON and looses the oData representation, which is important when it comes to timestamps and dates.

await sfsfSrv.get(data.$next)

martin.stenzig3 as I watched your reCAP talk about data replication performance from SF, you may have already encountered this problem?

Thanks,
Nico

[martinstenzig]

Nico,

I think that request is still open to get access to/use the next link to page through the server side information.

The 2 current problems are:

1. CAP filters out the next link and/or does not automatically page through the results.

2. The OData Connection does not allow you to provide URL Parameters like "paging=snapshot" and "dateFrom=1901-01-01"

I am presently using a clean REST call outside CAP or the SDK to to deal with the SF specialties and then bring it back into the standard framework. The one thing I was testing was to potentially use a REST type connection to overcome the challenges above and then apply the inofficial CAP OData data conversion function to apply to the result set to convert dates etc to the standard format.

Let me know if you have any better ideas!

[nicorunge]

Hi martin.stenzig3,

thanks for your prompt feedback.

1. This is what, at least from my understanding, the sap-cloud-sdk should be able to do already.
"By default, the SAP Cloud SDK automatically resolves all pages of a result-set if server-driven paging is encountered."
I tried to test by generating my own sap-cloud-sdk client, but so far I still have problems getting it to run at all.

2. That would be helpful! But I guess it's a bit tricky to implement, as the cql syntax should continue to work the same also work for non-external resources.

Can you give me a hint, how I can find the inofficial CAP OData data conversion function? This is exactly the piece I'm missing in my prototype! Then I could do my sf calls using executeHttpRequest and convert everything in a proper format afterwards.

Thanks!

Nico

[deekshasinha]

Hi nicorunge , The docs you have linked to are for the JAVA sdk. The JavaScript sdk does not provide support for $skipToken.

[nicorunge]

Arg, so the search is not respecting the previous chosen sdk on the start page... thanks for the hint deekshasinha!

So there is no build in way in the cloud sdk, where a skiptoken can be passed in (at least for the js sdk).

Everything has to be done manually by fetching the data using executeHttpRequest for each new skiptoken, and afterwards the data has to be deserialized somehow?! It is still unclear to me, how to get the data in the right oData Entity representation, when getting the result in JSON format.

[deekshasinha]

You are right, you can use the executeHttpRequest method to execute the request (adding the $skipToken) and deserializeEntity() to transform your response data in JSON format to an OData entity representation. Are you getting any errors when you call this?

With generated client you could potentially use the appendPath as shown here, but I assume you're not generating a client yourself.

[nicorunge]

Hi deekshasinha,

the deserializeEntity() function seems to require two parameters. The JSON data and a service entity representation (e.g. businessPartnerApi), which is used for the conversion.

How do I use this function in combination with executeHttpRequest, where I do not use a generated client, and therefore do not have a for example the businessPartnerApi representation?

BR,
Nico