POST'ing media using CDS odata v2 proxy and redirecting actual binary stream to 3rd party system

[tse-gdo]

Hi,

I am using the CAP model to expose a bunch of entities including attachments. These attachments are physically stored in back-end ECC and so I am using handlers to redirect the stream requests to back-end oData services. As a reference I used this guide cap-media-node which works well in reading and possibly writing entities. However my requirement is for oData v2 hence I am using the cds-odata-v2-adapter-proxy to redirect v2 requests to v4.

Read scenarios work well.

When trying to create a new attachment, oData v2 just requires a POST with binary stream and slug header with the required data. The proxy now takes this POST request and fires off:
1. a POST with json payload to create what i assume is a placeholder
2. a PUT/update request with the media stream to update the entity from 1 with the binary data.

Although this works okay in principle, my specific problem is that my back-end oData is the one that will create the entity for me and generate the required keys. And to do this create on back-end i need the binary stream which I only get from point 2 above. On point 1, I can mock return a success with some random keys which will take me to step 2 but then on the final success these random keys will be returned as opposed to the keys generated by my back-end oData service. Furthermore, if step 2 fails for any reason, the overall response is still a success as step 1 was a success. This of course is a problem as the user will assume both steps where completed successfully.

Looking at the v2 proxy code, maybe this could be enhanced to take into account the PUT request response? Also as the PUT request can technically return a response, use this response payload to overwrite the original create payload? Or any other idea to get around this?

[OliverKlemenz · Accepted solution]

I released version https://www.npmjs.com/package/@sap/cds-odata-v2-adapter-proxy/v/1.6.2, which is able to support this use case.

[OliverKlemenz]

Yes, exactly as I imagined it shall work. The reason is the following.

You call OData v2 Single POST, this goes through CDS OData V2 Adapter Proxy, which calls a POST followed by a PUT on OData v4 Service. OData v4 does not support single upload anymore, so the POST/PUT choreography is needed by OData v4. The handler then calls a single POST OData v2 call to the S4 system.

So everything’s ok from my side so far. Only thing is, that I would have expected that as POST holds the metadata and PUT the binary stream, both cannot be implemented empty, but I think best would be, to store the POST metadata temporarily in CREATE handler and then implement the UPDATE handler to combine the before stored metadata with the PUT stream to the new ODATA v2 POST request against the ABAP system…

It‘s kind of complex, but as ODATA v4 splits calls up, they need to be merged again… That‘s how it should work then…

[former_member833069]

Hello shubhmis,

We are trying the same thing that you were trying to do - Upload and Read attachments to Service Entry Sheet to an on-premise system. We are trying to refactor an existing UI5 application to use a CAP service instead of the oData service.

At this time, we are still not able to attach or read attachments.

Any help in this matter will be greatly appreciated. Thanks a lot.

[former_member833069]

Hello oliver.klemenz,

I am using the latest version of sap/cds-odata-v2-adapter-proxy.

When I try to upload a PDF file, it fails with the below error.

My code is as below. MimeType is resolving to 'application/pdf'var header = req.headers; var slug = header.slug; var result = await service.tx(req).send({ query: req.query, headers: { "Content-Type": req.data["MimeType"], "slug": slug } });

return result;

[cov2ap] - Proxy: Error: write EPIPE

at afterWriteDispatched (node:internal/stream_base_commons:160:15)

at writeGeneric (node:internal/stream_base_commons:151:3)

at Socket._writeGeneric (node:net:817:11)

at Socket._write (node:net:829:8)

at doWrite (node:internal/streams/writable:408:12)

at clearBuffer (node:internal/streams/writable:569:7)

at Socket.Writable.uncork (node:internal/streams/writable:348:7)

at connectionCorkNT (node:_http_outgoing:797:8)

at processTicksAndRejections (node:internal/process/task_queues:82:21) {

errno: -32,

code: 'EPIPE',

syscall: 'write'

}

[OliverKlemenz]

Hello Raj,

I think the issue is, that calling the service via CDS runtime like this:

service.tx(req).send({ query: req.query, headers: { "Content-Type": req.data["MimeType"], "slug": slug } });

does not support the handling for streams/binary, afaik. According to my assumption, CDS runtime does not consume the stream, that's why you get a "write EPIPE", as there is not target to write to.

Forwarding an upload scenario is not (yet) available via service interface.

johannesvogel, david.kunz2 can you maybe check, what's the status there, and if there are plans to support also this use-case?

You can implement the http call yourself, using your favorite http library (node-fetch, axios, request, etc. ...). Then you can handle the piping yourself, and it should work as expected. You only need to get the destination/credentials behind the service.

johannesvogel, david.kunz2 Is there a good way to re-use the service part, finding the destination, credentials, cloud connector, etc. but still call the http call yourself? Maybe the cloud-sdk for node.js needs to be used directly. Any guidance on that?

Best regards,

Oliver

[Shubham_M]

Hi oliver.klemenz,

Can you please provide me insight on how to use read stream in CAP. I want to call my backend get stream method and get the file downloaded.

[OliverKlemenz]

vansyckel Can you provide details, how to consume streams through service api.. via connect.to ?

[vansyckel]

Please see my comment above.

Best,
Sebastian

[Shubham_M]

Hi oliver.klemenz,

As mentioned earlier I have used the external S4 hana service in CAP, there is no element "Data" in the entity, create stream method in the backend already has one standard field where the data stream comes, without having to give any special element for binary stream. That's why the problem is.

Now I have got it working.

Below is the code I have used.

entity AttachmentSet as projection on external.AttachmentSet;
extend external.AttachmentSet with @title : 'AttachmentSet'{ Data: LargeBinary }
annotate AttachmentSet { @Core.ContentDisposition.Filename : FileName @Core.MediaType: MimeType Data; }But this does makes two calls :1. POST2. PUT

So I have to explicitly handle that in the "UPDATE" handler and pass a blank return.

this.on('UPDATE', 'AttachmentSet', async req => { return result = {} });But still if you can see if this approach is fine and why the PUT request is called .

[OliverKlemenz]

Yes, exactly as I imagined it shall work. The reason is the following.

You call OData v2 Single POST, this goes through CDS OData V2 Adapter Proxy, which calls a POST followed by a PUT on OData v4 Service. OData v4 does not support single upload anymore, so the POST/PUT choreography is needed by OData v4. The handler then calls a single POST OData v2 call to the S4 system.

So everything’s ok from my side so far. Only thing is, that I would have expected that as POST holds the metadata and PUT the binary stream, both cannot be implemented empty, but I think best would be, to store the POST metadata temporarily in CREATE handler and then implement the UPDATE handler to combine the before stored metadata with the PUT stream to the new ODATA v2 POST request against the ABAP system…

It‘s kind of complex, but as ODATA v4 splits calls up, they need to be merged again… That‘s how it should work then…

[Shubham_M]

Hi oliver.klemenz,

Is the patch not available for calling the service through a single POST call.

[OliverKlemenz]

Yes, the patch is long time available. When you use the latest version, you are on the safe side.

The following annotation shall be correct, assuming „Data“ is a valid element of the SES service entity.

annotate AttachmentSet {
@Core.IsMediaType
MimeType;

@Core.ContentDisposition.Filename: FileName
@Core.MediaType: MimeType
Data;
}

In another comment of another issue you mentioned the OData model as screenshot. I cannot see, that an „Data“ element exists.

To get the upload working with a single POST, you need to annotate an existing element with @Core.MediaType, like above. If Data does not exist, the annotation is simply ignored, and you will get the posted error message, as the single-POST magic is not applied.

I‘m wondering why AttachmentSet does not have something like a „Data“ element of type LargeBinary to which the data is written…

If such an element does not exist in backend service, you need to extend the projected entity Attachment by a Data element with @Core.MediaType annotation, so that single-POST upload can be triggered by CDS OData v2 Adapter Proxy

I strongly recommend to enable debug logs, as you then see the concrete OData v4 requests. To get it working correctly you need to have two request, one POST, that only creates the entity (CREATE handler called) and the second request a PUT to update the stream (UPDATE handler called).

I think your current concrete issue is, that Data is not a valid element and therefore the annotation is ignored and @Core.MediaType is not found… therefore the current error message…

[Shubham_M]

Hi oliver.klemenz

Using below code:

annotate AttachmentSet {
@Core.IsMediaType
MimeType;

@Core.ContentDisposition.Filename: FileName
@Core.MediaType: MimeType
Data;
}

gives me error

No payload deserializer available for resource kind 'ENTITY' and mime type 'image/png'

Also it doesn't fails at the gateway level rather fails on returning the results from CAP. I am already using try catch it doesn't go into that.

Also I tried to use below code :

annotate AttachmentSet { @Core.ContentDisposition.Filename : FileName @Core.MediaType : MimeType @Core.IsMediaType ServiceEntrySheetNo (Field in backend entity) }and it gives me error Property of Edm.Stream are not supported

[OliverKlemenz]

Ok, I think I understood your scenario...

Point 1: OData V4 does not allow to POST and send binary data in one call, therefore the first call will always have no binary data.

Point 2: Yes, I think this can be improved, that if the PUT fails, the overall status is failed... That makes sense... I will have a look...

The other use-case I need to get more details on. Changing the keys between POST and PUT is a little bit tricky. How would that happen? Would the PUT return the new keys in response body? The PUT normally is only for the blob data, so where to get the real keys from (standardized)?

[tse-gdo]

Thank you for the reply Oliver. If the PUT is allowed to respond with a json payload and this payload is used in place of "overwriteResponse.body" then that should hopefully solve the problem. I understand this is tricky. I am currently trying to hack something into the req._.res to see if that can be mapped correctly in the proxy code.

[OliverKlemenz]

I would try to change the function convertProxyResponse:

let statusCode = proxyRes.statusCode;
let headers = proxyRes.headers;
if (statusCode < 400 && req.overwriteResponse) {
  statusCode = req.overwriteResponse.statusCode;
  headers = {
    ...req.overwriteResponse.headers,
    ...headers};
}

and

let body = await parseProxyResponseBody(proxyRes, headers, req);
if (statusCode < 400 && req.overwriteResponse) {
  body = {
    ...req.overwriteResponse.body,
    ...body};
}

Could you have a look, if this addresses your two points, and you are able to overwrite the key in PUT call?

If that works for you, I prepare a new patch release...

[tse-gdo]

That seems to work perfectly. In the PUT, after successfully calling back-end, I just add the response in req._.res and everything is mapped correctly including overwriting keys and mapping it in odata metadata response.

let dataString = JSON.stringify({
    AttachmentID: resp.AttachmentID,
    Filename: resp.Filename,
    FileExtension: resp.FileExtension,
    MimeType: resp.MimeType
});

req._.res.setHeader('Content-Type', 'application/json');
req._.res.setHeader('Content-Length', dataString.length);
req._.res.end(dataString);

[OliverKlemenz]

Good to hear. I will prepare a new patch release tomorrow. Stay tuned...

[sujitsingh5191]

Hi oliver.klemenz/gdo-invokers,

I am trying to call my s4hana backend service for uploading attachments through CAP using edmx upload and exposing the entities as projection. In my catalog-service.js file I have defined the CREATE event handler for attachments and I am using the odata v2 adapter proxy 1.7.15 but i am getting payload deserialization error.

Can you please help me on this.

[OliverKlemenz]

Yes, I will try to help you. But I need more information on the request you are trying to perform. A payload deserialization error normally occurs, when the request body is not accepted by the OData server. E.g. a element is included, which is not known to the entity metadata or a value type is not correct.

Can you please:
- Try the latest version: https://www.npmjs.com/package/@sap/cds-odata-v2-adapter-proxy/v/1.8.4
- Provide the exact error message and the entity structure
- Provide the HTTP call information, you are trying to perform
- Try to activate debug log by setting node env variable XS_APP_LOG_LEVEL = debug to see the exact requests and responses

Thanks.

[Shubham_M]

Hi oliver.klemenz,

I am using the latest odata-v2-adapter-proxy 1.8.4

I am uploading the edmx file of the metadata of s4 hana system to cap and then having below code in CAP.

My all entities are working fine but I am getting a problem for Attachment

My code:

In catalog-service.js

this.on('CREATE', 'AttachmentSet', async req => { try { const srv = await cds.connect.to('SES') let result = await srv.transaction(req).send({ query: req.query })
return result } catch (error) { req.reject(400, error.innererror.response.body.error.message.value) }

});

In catalog-service.cds

entity AttachmentSet as projection on external.AttachmentSet;

URL:

POST:http://localhost:4004/v2/sap/opu/odata/cis2se1/SERVICE_ORDER_SRV/AttachmentSet

[OliverKlemenz]

Ok, thanks for the details. Yes, the POST of a binary file against a OData V2 Media-Type Entity enabled entity shall be possible.
Is your "CREATE" handler called? I guess not, as the attached error message already stops during request processing.

Can you please enabled debug logging by setting node env variable XS_APP_LOG_LEVEL = debug.

Can you then provide insights into the console.log, to see which request are done against the Node.js OData backend server?

In addition how does your entity look like? Is there an element existing, annotated by `@Core.MediaType`.

This annotation is necessary, to let the OData v2 Adapter Proxy do it's magic. If it's not part of the S4 metadata, you can try to add additional annotation via annotate keyword (cdl)

An simple CDS example looks as follows:

entity Header: cuid {
  @Core.MediaType: mediaType
  @Core.ContentDisposition.Filename: filename
  data: LargeBinary;
  @Core.IsMediaType
  mediaType: String;
  filename: String;
}

[OliverKlemenz]

Modeling details can be found here:

https://cap.cloud.sap/docs/guides/providing-services#media-data

[Shubham_M]

Hi oliver.klemenz,

1. Yes you are right. The CREATE handler is not called .

2. I will see the debug logs and post that here in some time.

3. Since this is a service in the backend ECC system and I have exposed this on CAP through edmx file. I have projected the entity as entity AttachmentSet as projection on external.AttachmentSet as stated earlier. So there is no element annotated by `@Core.MediaType`.

How can I extend the AttachmentSet as I feel '@Core.MediaType'. is an element level annotation.

I tried to use below code but still I am getting same error.

extend AttachmentSet with @Core.MediaType:mediaType;

[OliverKlemenz]

You can annotate an existing entity element like this:

annotate AttachmentSet {
    @Core.MediaType: 'image/png'
    data;
}

If you have an element in AttachmentSet, that can take the media type I would recommend to use:

annotate AttachmentSet {
    @Core.MediaType: mediaType
    data;
}

[Shubham_M]

Hi oliver.klemenz,

Thanks for the prompt response!

Now at least I can call the backend create stream and upload the attachments. But while returning result back I get below error.

Code for annotation extension:

entity AttachmentSet as projection on external.AttachmentSet;
annotate AttachmentSet { @Core.IsMediaType @Core.MediaType: MimeType MimeType; @Core.ContentDisposition.Filename: FileName MimeType; }Code in service.jsthis.on('CREATE', 'AttachmentSet', async req => { try { const srv = await cds.connect.to('SES') return result = await srv.transaction(req).send({ query: req.query, headers: { "Content-Type": req.data["MimeType"], "slug": req.data["FileName"] } })
}Error:

[OliverKlemenz]

I think the error message is raised by ABAP backend (from SAP Gateway).
The issue could be, that the wrong field (MimeType) is tried to be written with binary,

You have the annotation like this:

annotate AttachmentSet {
    @Core.IsMediaType
    @Core.MediaType: MimeType
    MimeType;

    @Core.ContentDisposition.Filename: FileName
    MimeType;
}

I would have expected it like this (there needs to be a binary property, e.g. data, that needs to be annotated):

annotate AttachmentSet {
    @Core.IsMediaType
    MimeType;

    @Core.ContentDisposition.Filename: FileName
    @Core.MediaType: MimeType
    Data;
}

Can you check, if you can apply the annotation as proposed?

In addition you can try/catch errors in your CREATE handler implementation, so that you can check, if the ABAP backend call is the one that is failing.

[OliverKlemenz]

Please also keep in mind, that the OData v2 request is split up into 2 OData v4 requests
- 1st OData to create the Entity AttachmentSet (metadata only no binary/stream) => CREATE handler is called
- 2nd OData request to update the binary data element of AttachmentSet (includes stream) => UPDATE handler is called

Therefore I think the query to the SES service needs to be implemented in the UPDATE handler, when the "data" element is updated, holding the stream query...

Can you check?

[OliverKlemenz]

And in addition I'm not sure, if CDS runtime can proxy the req.query like you try it for a stream scenario:

const srv = await cds.connect.to('SES')
return result = await srv.transaction(req).send({ query: req.query, headers: { "Content-Type": req.data["MimeType"], "slug": req.data["FileName"] } })

vansyckel Can you maybe comment on that scenario, if this is supported by CDS Node.js runtime out-of-the-box for binary/stream upload queries?

Otherwise the HTTP request to ABAP need to be build up manually, piping the stream of req.

[vansyckel]

Hi all,

Streaming for remote services is not yet supported. You could implement yourself, though. Please see Consuming Services for details on how to interact with remote services. Your custom on handler would need to return a Node.js stream object the same way our database service implementation does.

Best,
Sebastian

[Shubham_M]

Hi vansyckel,

The problem is I am not having any properties in backend which returns stream values. So the problem is how do i return the Node.js stream.

[OliverKlemenz]

If the backend request is an OData v2 service, you don't need to specify an element, IMO, but you directly access stream via $value on entity.

So something like this should return a stream, hopefully:
GET /sap/opu/odata/cis2se1/SERVICEORDERSRV/AttachmentSet(<key>)/$value

[Shubham_M]

Hi oliver.klemenz,

I am able to send this request GET /sap/opu/odata/cis2se1/SERVICEORDERSRV/AttachmentSet(<key>)/$value

to backend but I am not able to get the stream values from the backend. The backend sends the stream correctly. But the variable result doesn't show the values anywhere.

[OliverKlemenz]

How do you call the get request? Which library do you use? You should not await the HTTP call, but use the stream handle and return it to cds runtime as structure { value: stream }

[Shubham_M]

Hi oliver.klemenz

Yes I was using the await call.

this.on('READ', 'AttachmentSet', async req => { try { const srv = await cds.connect.to('SES') if (req.data.DocumentId !=null){ var str = '/\AttachmentSet(DocumentId=\''+req.data.DocumentId+ '\',ServiceEntrySheetNo=\''+ req.data.ServiceEntrySheetNo+'\',PONumber=\''+req.data.PONumber+'\',Item=\''+req.data.Item+'\')/$value' let result = await srv.get(str) return result }else{ let result = await srv.transaction(req).send({ query: req.query, }) return result

Any references how I can I use the stream handle .

[OliverKlemenz]

I think you need to do something yourself, e.g. using request Node.js library (https://github.com/request/request#streaming), but any other http library is good as well). I think you cannot use cds.connect.to("SES"), at least for calling, as according to vansyckel streaming is not implemented, so you need to do this on your own (for now). Maybe you can get the url and credentials from CDS somehow, otherwise you need to call the destination service. You can try to use the sap-cloud-sdk for JS (https://sap.github.io/cloud-sdk/docs/js/getting-started), to easen this task (otherwise you need to call the connectivity service yourself as well)... vansyckel: Is it possible to get the URL and Authentication from cds.connect.to("SES"), so not making the concrete call, but just getting the necessary info to perform a call yourself (maybe these are 2 requirements, support for stream through CDS connected service, and getting the connection details from a connected service from CDS).

Here's the code (pseudo code):

const request = require('request');

this.on("READ", "AttachmentSet", async (req) => {
  if (context._.odataReq._url.pathname.endsWith("/Data") || context._.odataReq._url.pathname.endsWith("/Data/$value")) {
    if (req.data.DocumentId !== null && req.data.ServiceEntrySheetNo !== null && req.data.PONumber !== null && req.data.Item !== null) {
      const backendUrl = "<path-to-backend-system>"; // Use destination service to retrieve this url and call through connectivity service (use best sap-cloud-sdk)
      const entityPath = `AttachmentSet(DocumentId='${ req.data.DocumentId }',ServiceEntrySheetNo='${ req.data.ServiceEntrySheetNo }',PONumber='${ req.data.PONumber }',Item='${ req.data.Item }')/$value`;
      const stream = request.get(backendUrl + "/" + entityPath, {
        // auth: {} // add your authorization here, credentials to be retrieved from destination service 
      });
      return {
        value: stream,
        "*@odata.mediaContentType": "application/pdf"
      };
    }
  }
  req.error("Key information not provided");
});

[Shubham_M]

Hi oliver.klemenz,

Thanks for your response!

I would try to implement this and see how it goes.

Is there any road map from SAP when streaming of media resources for external services would be supported .

[OliverKlemenz]

vansyckel Can you say something, if and when streaming for external services is on CAPs roadmap?

[vansyckel]

Hi,

Streaming from remote is not on the 2022 roadmap. Please also note that Node.js - Custom Streaming describes a bit of a workaround and may change in the future.

Best,
Sebastian

[Shubham_M]

Hi oliver.klemenz,

I tried to implement the read stream through calling the destination. I am able to get the stream locally but when I deploy the application

it gives me below error.

NTERNAL ERROR] TypeError: stream.push is not a function

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Request.<anonymous> (/home/vcap/app/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:322:14)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Request.emit (node:events:402:35)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Request.onRequestError (/home/vcap/app/node_modules/request/request.js:877:8)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at ClientRequest.emit (node:events:390:28)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Socket.socketErrorListener (node:_http_client:447:9)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at Socket.emit (node:events:390:28)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at emitErrorNT (node:internal/streams/destroy:157:8)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at emitErrorCloseNT (node:internal/streams/destroy:122:3)

2022-02-02T17:20:47.073+0000 [APP/PROC/WEB/0] ERR at processTicksAndRejections (node:internal/process/task_queues:83:21)

I think this has something to do with the require(request).

[OliverKlemenz]

Yes, it looks as if your handler implementation does not return a valid stream object (that has push function) back to the CDS generic OData read handler.

Can you try setting up debug mode in your deployed version following this guide:
https://blogs.sap.com/2021/06/11/set-up-remote-debugging-to-diagnose-cap-applications-node.js-stack-...

Then you are maybe able to directly see, what is happening there..

=====
Excerpt from: @sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js

let result = await tx.dispatch(req)
const streamObj = result[0]

const stream = streamObj.value

if (stream) {
  stream.on('error', () => {
    stream.removeAllListeners('error')
    // stream.destroy() does not end stream in node 10 and 12
    stream.push(null) // --> Here is breaks, as result does not return correct stream object
  })

}

[Shubham_M]

Hi oliver.klemenz,

Thank you!

I will try to debug the deployed version.

But I have one doubt the url returns valid stream when running locally.