POST'ing media using CDS odata v2 proxy and redirecting actual binary stream to 3rd party system

[tse-gdo]

Hi,

I am using the CAP model to expose a bunch of entities including attachments. These attachments are physically stored in back-end ECC and so I am using handlers to redirect the stream requests to back-end oData services. As a reference I used this guide cap-media-node which works well in reading and possibly writing entities. However my requirement is for oData v2 hence I am using the cds-odata-v2-adapter-proxy to redirect v2 requests to v4.

Read scenarios work well.

When trying to create a new attachment, oData v2 just requires a POST with binary stream and slug header with the required data. The proxy now takes this POST request and fires off:
1. a POST with json payload to create what i assume is a placeholder
2. a PUT/update request with the media stream to update the entity from 1 with the binary data.

Although this works okay in principle, my specific problem is that my back-end oData is the one that will create the entity for me and generate the required keys. And to do this create on back-end i need the binary stream which I only get from point 2 above. On point 1, I can mock return a success with some random keys which will take me to step 2 but then on the final success these random keys will be returned as opposed to the keys generated by my back-end oData service. Furthermore, if step 2 fails for any reason, the overall response is still a success as step 1 was a success. This of course is a problem as the user will assume both steps where completed successfully.

Looking at the v2 proxy code, maybe this could be enhanced to take into account the PUT request response? Also as the PUT request can technically return a response, use this response payload to overwrite the original create payload? Or any other idea to get around this?

[OliverKlemenz]

Ok, I think I understood your scenario...

Point 1: OData V4 does not allow to POST and send binary data in one call, therefore the first call will always have no binary data.

Point 2: Yes, I think this can be improved, that if the PUT fails, the overall status is failed... That makes sense... I will have a look...

The other use-case I need to get more details on. Changing the keys between POST and PUT is a little bit tricky. How would that happen? Would the PUT return the new keys in response body? The PUT normally is only for the blob data, so where to get the real keys from (standardized)?

[tse-gdo]

Thank you for the reply Oliver. If the PUT is allowed to respond with a json payload and this payload is used in place of "overwriteResponse.body" then that should hopefully solve the problem. I understand this is tricky. I am currently trying to hack something into the req._.res to see if that can be mapped correctly in the proxy code.

[OliverKlemenz]

I would try to change the function convertProxyResponse:

let statusCode = proxyRes.statusCode;
let headers = proxyRes.headers;
if (statusCode < 400 && req.overwriteResponse) {
  statusCode = req.overwriteResponse.statusCode;
  headers = {
    ...req.overwriteResponse.headers,
    ...headers};
}

and

let body = await parseProxyResponseBody(proxyRes, headers, req);
if (statusCode < 400 && req.overwriteResponse) {
  body = {
    ...req.overwriteResponse.body,
    ...body};
}

Could you have a look, if this addresses your two points, and you are able to overwrite the key in PUT call?

If that works for you, I prepare a new patch release...

[tse-gdo]

That seems to work perfectly. In the PUT, after successfully calling back-end, I just add the response in req._.res and everything is mapped correctly including overwriting keys and mapping it in odata metadata response.

let dataString = JSON.stringify({
    AttachmentID: resp.AttachmentID,
    Filename: resp.Filename,
    FileExtension: resp.FileExtension,
    MimeType: resp.MimeType
});

req._.res.setHeader('Content-Type', 'application/json');
req._.res.setHeader('Content-Length', dataString.length);
req._.res.end(dataString);

[OliverKlemenz]

Good to hear. I will prepare a new patch release tomorrow. Stay tuned...

[sujitsingh5191]

Hi oliver.klemenz/gdo-invokers,

I am trying to call my s4hana backend service for uploading attachments through CAP using edmx upload and exposing the entities as projection. In my catalog-service.js file I have defined the CREATE event handler for attachments and I am using the odata v2 adapter proxy 1.7.15 but i am getting payload deserialization error.

Can you please help me on this.

[OliverKlemenz]

Yes, I will try to help you. But I need more information on the request you are trying to perform. A payload deserialization error normally occurs, when the request body is not accepted by the OData server. E.g. a element is included, which is not known to the entity metadata or a value type is not correct.

Can you please:
- Try the latest version: https://www.npmjs.com/package/@sap/cds-odata-v2-adapter-proxy/v/1.8.4
- Provide the exact error message and the entity structure
- Provide the HTTP call information, you are trying to perform
- Try to activate debug log by setting node env variable XS_APP_LOG_LEVEL = debug to see the exact requests and responses

Thanks.

[Shubham_M]

Hi oliver.klemenz,

I am using the latest odata-v2-adapter-proxy 1.8.4

I am uploading the edmx file of the metadata of s4 hana system to cap and then having below code in CAP.

My all entities are working fine but I am getting a problem for Attachment

My code:

In catalog-service.js

this.on('CREATE', 'AttachmentSet', async req => { try { const srv = await cds.connect.to('SES') let result = await srv.transaction(req).send({ query: req.query })
return result } catch (error) { req.reject(400, error.innererror.response.body.error.message.value) }

});

In catalog-service.cds

entity AttachmentSet as projection on external.AttachmentSet;

URL:

POST:http://localhost:4004/v2/sap/opu/odata/cis2se1/SERVICE_ORDER_SRV/AttachmentSet

[OliverKlemenz]

Ok, thanks for the details. Yes, the POST of a binary file against a OData V2 Media-Type Entity enabled entity shall be possible.
Is your "CREATE" handler called? I guess not, as the attached error message already stops during request processing.

Can you please enabled debug logging by setting node env variable XS_APP_LOG_LEVEL = debug.

Can you then provide insights into the console.log, to see which request are done against the Node.js OData backend server?

In addition how does your entity look like? Is there an element existing, annotated by `@Core.MediaType`.

This annotation is necessary, to let the OData v2 Adapter Proxy do it's magic. If it's not part of the S4 metadata, you can try to add additional annotation via annotate keyword (cdl)

An simple CDS example looks as follows:

entity Header: cuid {
  @Core.MediaType: mediaType
  @Core.ContentDisposition.Filename: filename
  data: LargeBinary;
  @Core.IsMediaType
  mediaType: String;
  filename: String;
}

[OliverKlemenz]

Modeling details can be found here:

https://cap.cloud.sap/docs/guides/providing-services#media-data

[Shubham_M]

Hi oliver.klemenz,

1. Yes you are right. The CREATE handler is not called .

2. I will see the debug logs and post that here in some time.

3. Since this is a service in the backend ECC system and I have exposed this on CAP through edmx file. I have projected the entity as entity AttachmentSet as projection on external.AttachmentSet as stated earlier. So there is no element annotated by `@Core.MediaType`.

How can I extend the AttachmentSet as I feel '@Core.MediaType'. is an element level annotation.

I tried to use below code but still I am getting same error.

extend AttachmentSet with @Core.MediaType:mediaType;

[OliverKlemenz]

You can annotate an existing entity element like this:

annotate AttachmentSet {
    @Core.MediaType: 'image/png'
    data;
}

If you have an element in AttachmentSet, that can take the media type I would recommend to use:

annotate AttachmentSet {
    @Core.MediaType: mediaType
    data;
}

[Shubham_M]

Hi oliver.klemenz,

Thanks for the prompt response!

Now at least I can call the backend create stream and upload the attachments. But while returning result back I get below error.

Code for annotation extension:

entity AttachmentSet as projection on external.AttachmentSet;
annotate AttachmentSet { @Core.IsMediaType @Core.MediaType: MimeType MimeType; @Core.ContentDisposition.Filename: FileName MimeType; }Code in service.jsthis.on('CREATE', 'AttachmentSet', async req => { try { const srv = await cds.connect.to('SES') return result = await srv.transaction(req).send({ query: req.query, headers: { "Content-Type": req.data["MimeType"], "slug": req.data["FileName"] } })
}Error:

[OliverKlemenz]

I think the error message is raised by ABAP backend (from SAP Gateway).
The issue could be, that the wrong field (MimeType) is tried to be written with binary,

You have the annotation like this:

annotate AttachmentSet {
    @Core.IsMediaType
    @Core.MediaType: MimeType
    MimeType;

    @Core.ContentDisposition.Filename: FileName
    MimeType;
}

I would have expected it like this (there needs to be a binary property, e.g. data, that needs to be annotated):

annotate AttachmentSet {
    @Core.IsMediaType
    MimeType;

    @Core.ContentDisposition.Filename: FileName
    @Core.MediaType: MimeType
    Data;
}

Can you check, if you can apply the annotation as proposed?

In addition you can try/catch errors in your CREATE handler implementation, so that you can check, if the ABAP backend call is the one that is failing.

[OliverKlemenz]

Please also keep in mind, that the OData v2 request is split up into 2 OData v4 requests
- 1st OData to create the Entity AttachmentSet (metadata only no binary/stream) => CREATE handler is called
- 2nd OData request to update the binary data element of AttachmentSet (includes stream) => UPDATE handler is called

Therefore I think the query to the SES service needs to be implemented in the UPDATE handler, when the "data" element is updated, holding the stream query...

Can you check?

[OliverKlemenz]

And in addition I'm not sure, if CDS runtime can proxy the req.query like you try it for a stream scenario:

const srv = await cds.connect.to('SES')
return result = await srv.transaction(req).send({ query: req.query, headers: { "Content-Type": req.data["MimeType"], "slug": req.data["FileName"] } })

vansyckel Can you maybe comment on that scenario, if this is supported by CDS Node.js runtime out-of-the-box for binary/stream upload queries?

Otherwise the HTTP request to ABAP need to be build up manually, piping the stream of req.

[vansyckel]

Hi all,

Streaming for remote services is not yet supported. You could implement yourself, though. Please see Consuming Services for details on how to interact with remote services. Your custom on handler would need to return a Node.js stream object the same way our database service implementation does.

Best,
Sebastian

[Shubham_M]

Hi vansyckel,

The problem is I am not having any properties in backend which returns stream values. So the problem is how do i return the Node.js stream.

[OliverKlemenz]

If the backend request is an OData v2 service, you don't need to specify an element, IMO, but you directly access stream via $value on entity.

So something like this should return a stream, hopefully:
GET /sap/opu/odata/cis2se1/SERVICEORDERSRV/AttachmentSet(<key>)/$value

[Shubham_M]

Hi oliver.klemenz,

I am able to send this request GET /sap/opu/odata/cis2se1/SERVICEORDERSRV/AttachmentSet(<key>)/$value

to backend but I am not able to get the stream values from the backend. The backend sends the stream correctly. But the variable result doesn't show the values anywhere.

[OliverKlemenz]

How do you call the get request? Which library do you use? You should not await the HTTP call, but use the stream handle and return it to cds runtime as structure { value: stream }

[Shubham_M]

Hi oliver.klemenz

Yes I was using the await call.

this.on('READ', 'AttachmentSet', async req => { try { const srv = await cds.connect.to('SES') if (req.data.DocumentId !=null){ var str = '/\AttachmentSet(DocumentId=\''+req.data.DocumentId+ '\',ServiceEntrySheetNo=\''+ req.data.ServiceEntrySheetNo+'\',PONumber=\''+req.data.PONumber+'\',Item=\''+req.data.Item+'\')/$value' let result = await srv.get(str) return result }else{ let result = await srv.transaction(req).send({ query: req.query, }) return result

Any references how I can I use the stream handle .

[OliverKlemenz]

I think you need to do something yourself, e.g. using request Node.js library (https://github.com/request/request#streaming), but any other http library is good as well). I think you cannot use cds.connect.to("SES"), at least for calling, as according to vansyckel streaming is not implemented, so you need to do this on your own (for now). Maybe you can get the url and credentials from CDS somehow, otherwise you need to call the destination service. You can try to use the sap-cloud-sdk for JS (https://sap.github.io/cloud-sdk/docs/js/getting-started), to easen this task (otherwise you need to call the connectivity service yourself as well)... vansyckel: Is it possible to get the URL and Authentication from cds.connect.to("SES"), so not making the concrete call, but just getting the necessary info to perform a call yourself (maybe these are 2 requirements, support for stream through CDS connected service, and getting the connection details from a connected service from CDS).

Here's the code (pseudo code):

const request = require('request');

this.on("READ", "AttachmentSet", async (req) => {
  if (context._.odataReq._url.pathname.endsWith("/Data") || context._.odataReq._url.pathname.endsWith("/Data/$value")) {
    if (req.data.DocumentId !== null && req.data.ServiceEntrySheetNo !== null && req.data.PONumber !== null && req.data.Item !== null) {
      const backendUrl = "<path-to-backend-system>"; // Use destination service to retrieve this url and call through connectivity service (use best sap-cloud-sdk)
      const entityPath = `AttachmentSet(DocumentId='${ req.data.DocumentId }',ServiceEntrySheetNo='${ req.data.ServiceEntrySheetNo }',PONumber='${ req.data.PONumber }',Item='${ req.data.Item }')/$value`;
      const stream = request.get(backendUrl + "/" + entityPath, {
        // auth: {} // add your authorization here, credentials to be retrieved from destination service 
      });
      return {
        value: stream,
        "*@odata.mediaContentType": "application/pdf"
      };
    }
  }
  req.error("Key information not provided");
});

[Shubham_M]

Hi oliver.klemenz,

Thanks for your response!

I would try to implement this and see how it goes.

Is there any road map from SAP when streaming of media resources for external services would be supported .

[OliverKlemenz]

vansyckel Can you say something, if and when streaming for external services is on CAPs roadmap?

[vansyckel]

Hi,

Streaming from remote is not on the 2022 roadmap. Please also note that Node.js - Custom Streaming describes a bit of a workaround and may change in the future.

Best,
Sebastian

[Shubham_M]

Hi oliver.klemenz,

I tried to implement the read stream through calling the destination. I am able to get the stream locally but when I deploy the application

it gives me below error.

NTERNAL ERROR] TypeError: stream.push is not a function

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Request.<anonymous> (/home/vcap/app/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:322:14)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Request.emit (node:events:402:35)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Request.onRequestError (/home/vcap/app/node_modules/request/request.js:877:8)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at ClientRequest.emit (node:events:390:28)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Socket.socketErrorListener (node:_http_client:447:9)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Socket.emit (node:events:390:28)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at emitErrorNT (node:internal/streams/destroy:157:8)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at emitErrorCloseNT (node:internal/streams/destroy:122:3)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at processTicksAndRejections (node:internal/process/task_queues:83:21)

I think this has something to do with the require(request).

[OliverKlemenz]

Yes, it looks as if your handler implementation does not return a valid stream object (that has push function) back to the CDS generic OData read handler.

Can you try setting up debug mode in your deployed version following this guide:
https://blogs.sap.com/2021/06/11/set-up-remote-debugging-to-diagnose-cap-applications-node.js-stack-...

Then you are maybe able to directly see, what is happening there..

=====
Excerpt from: @sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js

let result = await tx.dispatch(req)
const streamObj = result[0]

const stream = streamObj.value

if (stream) {
  stream.on('error', () => {
    stream.removeAllListeners('error')
    // stream.destroy() does not end stream in node 10 and 12
    stream.push(null) // --> Here is breaks, as result does not return correct stream object
  })

}

[Shubham_M]

Hi oliver.klemenz,

Thank you!

I will try to debug the deployed version.

But I have one doubt the url returns valid stream when running locally.