"Failed to load destination" from within 'cds.xt.SaasProvisioningService' implementation

[pieterjanssens]

I would think that a binding to a destination service suffices to be able to resolve a subaccount level destination. There seems to be some bug or issue in my implementation failing it to work.

cds.requires:

"CFAPI": {
        "kind": "rest",
        "credentials": {
          "destination": "CFAPI"
        },
        "destinationOptions": {
          "selectionStrategy": "alwaysProvider"
        }
      }

cds server.js:

const proxy = require('@sap/cds-odata-v2-adapter-proxy')
const cds = require('@sap/cds')

cds.once('bootstrap', (app) => {
  app.use(proxy())
  app.get('/favicon.ico', (req, res) => res.status(204))
})

cds.on('served', async () => {
  const { 'cds.xt.SaasProvisioningService': provisioning } = cds.services
  await provisioning.prepend(() => {
    provisioning.impl(require('./saas'))
  })
})

module.exports = cds.server 

Minimal reproducible saas.js (the provisioning implementation):

const cds = require('@sap/cds')
const appEnv = JSON.parse(process.env.VCAP_APPLICATION)

const cfApi = await cds.connect.to('CFAPI')
const domainResponse = await cfApi.get(
   `/v3/domains?names=${/\.(.*)/gm.exec(appEnv.application_uris[0])[1]}`
)
console.log(domainResponse)

The error I'm facing:

 2023-02-14T22:57:12.86+0100 [APP/PROC/WEB/0] OUT [cds] - PUT /-/cds/saas-provisioning/tenant/2058684d-da95-4200-a655-92c77c751bcd
   2023-02-14T22:57:12.87+0100 [APP/PROC/WEB/0] OUT YO, NEW SUBSCRIBER HERE:  2058684d-da95-4200-a655-92c77c751bcd
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR [remote] - Error: Error during request to remote service:
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR Failed to load destination.
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at run (/home/vcap/deps/0/node_modules/@sap/cds/libx/_runtime/remote/utils/client.js:312:31)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async RemoteService.<anonymous> (/home/vcap/deps/0/node_modules/@sap/cds/libx/_runtime/remote/Service.js:234:20)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async next (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:75:17)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async RemoteService.handle (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:73:10)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async SaasProvisioningService.<anonymous> (/home/vcap/app/srv/saas.js:100:28)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async next (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:75:17)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async SaasProvisioningService.handle (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:73:10)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async module.exports (/home/vcap/deps/0/node_modules/@sap/cds/libx/rest/middleware/update.js:23:16) {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR statusCode: 502,
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR reason: {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR message: 'Error during request to remote service: \n' +
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR 'Failed to load destination. Caused by: Could not fetch client credentials token for service of type "destination". Caused by: undefined',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR request: {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR method: 'GET',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR url: '/v3/domains?names=cfapps.eu30.hana.ondemand.com',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR headers: {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR accept: 'application/json,text/plain',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR 'x-correlation-id': '1e9b6f8d-fd40-44b5-46ea-755cf22d0cb4'
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR }
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR },
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR correlationId: '1e9b6f8d-fd40-44b5-46ea-755cf22d0cb4'
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR }
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR }
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR [cds] - Error: Error during request to remote service:
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR Failed to load destination.
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at run (/home/vcap/deps/0/node_modules/@sap/cds/libx/_runtime/remote/utils/client.js:312:31)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async RemoteService.<anonymous> (/home/vcap/deps/0/node_modules/@sap/cds/libx/_runtime/remote/Service.js:234:20)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async next (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:75:17)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async RemoteService.handle (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:73:10)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async SaasProvisioningService.<anonymous> (/home/vcap/app/srv/saas.js:100:28)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async next (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:75:17)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async SaasProvisioningService.handle (/home/vcap/deps/0/node_modules/@sap/cds/lib/srv/srv-dispatch.js:73:10)
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR at async module.exports (/home/vcap/deps/0/node_modules/@sap/cds/libx/rest/middleware/update.js:23:16) {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR statusCode: 502,
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR reason: {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR message: 'Error during request to remote service: \n' +
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR 'Failed to load destination. Caused by: Could not fetch client credentials token for service of type "destination". Caused by: undefined',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR request: {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR method: 'GET',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR url: '/v3/domains?names=cfapps.eu30.hana.ondemand.com',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR headers: {
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR accept: 'application/json,text/plain',
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR 'x-correlation-id': '1e9b6f8d-fd40-44b5-46ea-755cf22d0cb4'
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR }
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR },
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR correlationId: '1e9b6f8d-fd40-44b5-46ea-755cf22d0cb4'
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR }
   2023-02-14T22:57:13.02+0100 [APP/PROC/WEB/0] ERR }

The connect.to was working at a certain point, but I am not succeeding in resolving it since it started failing.

As you can see in the sample below, if I use the code within "TEST BEGIN" and "TEST END", the destination can get resolved and the HTTP request works. Do note, to make this work I had to remove the 'destinationOptions'.

const proxy = require('@sap/cds-odata-v2-adapter-proxy')
const cds = require('@sap/cds')

cds.once('bootstrap', (app) => {
  app.use(proxy())
  app.get('/favicon.ico', (req, res) => res.status(204))
})

cds.on('served', async () => {<br><br>// TEST BEGIN
  const cfApi = await cds.connect.to('CFAPI')
  const appEnv = JSON.parse(process.env.VCAP_APPLICATION)
  const domainResponse = await cfApi.get(
    `/v3/domains?names=${/\.(.*)/gm.exec(appEnv.application_uris[0])[1]}`
  )
  console.log(domainResponse)
  return<br>// TEST END

  const { 'cds.xt.SaasProvisioningService': provisioning } = cds.services

  await provisioning.prepend(() => {
    provisioning.impl(require('./saas'))
  })
})

module.exports = cds.server<br>

[Arley]

Hi piejanssens,

The CDS version 6.8.1 was published recently. Please upgrade and let me know whether the upgrade solves the issue.

[Arley · Accepted solution]

Hi piejanssens and frank.essenberger1 ,

In the CAP CDS Runtime team, we have recently resolved an issue regarding the destination lookup behavior that seems to be related to the problem you described.

The issue was related to the selectionStrategy property. It was discovered that although CAP did provide the selection strategy of destination options to the SAP Cloud SDK, it was not in the expected format. This caused issues in resolving the destinations after starting to use the new major version 2 of the SAP Cloud SDK.

The fix is included in the upcoming CDS version 6.8.0.

Kind regards,
Arley Triana Morin

[frank_essenberger1]

Hi Pieter,

you are correct that a binding to the destination service should be sufficient. However, we inherited some bad practice from Java SDK that we get the tenant id from a binding to the XSUAA. We have a backlog item to remove this. Could you add a binding to a XSUAA service to the application to check if the error disappears?

If so I will link this question to our backlog item and bring this up in the next planning that we do not postpone working on the issue.

Best

Frank

[pieterjanssens]

My workaround/solution is to explicitly create a cds transaction based on the tenant ID of the provider subaccount.

I notice the cds.context of the incoming requests upon subscribing was referring to a tenant ID "sap-provisioning".

So that made me think, what if it's due to this specific context that the destination resolution fails. I then tried to manually set the context by passing the tenant ID (subdomain it seems to be... in this case) of my provider application and now it works:

const appEnv = JSON.parse(process.env.VCAP_APPLICATION)
const response = await cfApi.tx({ tenant: appEnv.organization_name }).get(`/v3/domains?names=...`)

[nicorunge]

Hi Frank,

two weeks ago, I stumbled across the same error message. The error appeared, although there was a xsuaa binding in place. I could get rid of the error, by implementing a custom dependency handler (slack & scn post), also thinking my previous attempt was not correct.

BR
Nico

[frank_essenberger1]

Hm, I missed Pieters comment on the TestEnd and TestStart and that the `alwaysProvider` options seems to make a difference. This is strange because if the option is there we should skip checking the subscriber account and go for the provider only - I assume the destination is only maintained in this subaccount so I am confused why the destination can be resolved when the subscriber is checked first.

Perhaps you can make a check and call the SDKs method `getDestination` directly and pass the options and see if it is working - this would help me to understand the reason for that - if the getDestination works as expected we need to involve the coleagues from CAP.

[frank_essenberger1]

@ piejanssens Great that you found at least a work around. Sorry for the inconvenience on your side. From your explanation I get the impression that the CDS/CAP layer does not provide the "right" information to the SDK and therefore the fetching fails - it makes sense that the fetching of a service token (client credentials grant) fails if one tries to do that for the wrong tenant. I will ping the colleagues to see how the tenant is propagated. I always assumed that they just hand over a sign JWT where we extract the tenant from but if this explicit setting makes a different they seem to use a different API.

I am not sure yet but perhaps this issue is also related to this one: https://github.com/SAP/cloud-sdk-js/issues/3067. We have an additional glitch that if you pass a subscriber JWT we try to get a service token for that account, even if you select 'onlyProvider' and the JWT is therefore irrelevant. However, from the issue we found out that CAP is always providing the JWT no matter what selection strategy is used. We have a ticket to make the whole token retrieval lazy so that a service token is only fetched when it is actually needed. The assumption would be that if you set the tenant explicitly they do not provide the subscriber JWT anymore and therefore it works. I hope we have time to implement the lazy loading soon because it will fix different issues regardless if it is related to this one.

Best

Frank

[pieterjanssens]

Hi Frank, I'd appreciate if you could ping the CAP folks. It looks as if the destination resolution fails because the tenant context is neither the provider or the subscriber (in the 'cds.xt.SaasProvisioningService' implementation the callbacks are called from the saas-registry service and this has tenant id "sap-provisioning" in the context). So I can't tell if it is related as that issue, but in either case the 'alwaysProvider' is not respected by cds.

[frank_essenberger1]

I have reached out to the colleagues and hope they will comment on the issue soon.