REST Adapter in PI/PO: Enhanced XML/JSON Conversion

[vadimklimov]

Update on 28.01.2016

Recently SCN document PI REST Adapter – JSON to XML conversion (a part of series describing SAP REST adapter functionality and technical capabilities) has been updated and now contains information on enhanced XML/JSON conversion and a demo example on this topic.

SAP Help documentation (currently only found in help materials for release SAP NetWeaver 7.31) now contains description of enhanced XML/JSON conversion - refer to Converting XML/JSON - Advanced Adapter Engine - SAP Library.

Both references have been added to a section "Links" of this blog.

One of features of SAP standard REST adapter, is XML/JSON conversion - that surely makes sense, considering that internal processing in SAP PI/PO is done in XML format on one hand, and JSON format is de-facto format when dealing with REST architectural style on the other hand. Looking into recent SCN forum threads and questions raised about a REST adapter, it can be concluded that generation of JSON output for a processed XML message payload is not always clear and may be misleading. SAP actively enhances functionality of a REST adapter – customization and feature-enrichment of JSON processing being one of actively contributed areas. Many of such features have been documented in SAP Help materials, but one of quite powerful and flexible functionalities – namely, enhanced XML/JSON conversion – was only briefly mentioned in a SAP Note 2175218. In this blog, I would like to demonstrate usage of this functionality and provide details about valid parameterization.

Internally, a REST adapter makes use of 3rd party Jettison library for JSON processing tasks. In standard configuration, REST adapter relies on default conversion logic implemented in Jettison processor, which does not correlate or take into consideration payload elements properties as defined in a corresponding message type, but has its own optimization and type derivation mechanisms that are based on nature of the value of a processed XML document's element rather than XSD schema of a processed message. As a result, this conversion may sometimes result in unobvious output – here are just few examples which are commonly faced:

• If an XML element was defined as an array, but only contains one item in converted XML payload, Jettison processor will likely convert it to a non-array type;


• If an XML element was defined as a String, but only contains numeric value in converted XML payload, Jettison processor will likely convert it to an integer type.

In some use cases, this kind of improper type conversion may be unacceptable – and this is where enhanced XML/JSON conversion parameterization helps solving a faced problem.

An idea behind enhanced XML/JSON conversion functionality introduced with the SAP Note 2175218, is explicit instruction of JSON processor on how to treat particular XML elements. Let us examine this functionality based on a practical example.

 

Below is definition of a message type used for a response message in a synchronous scenario, where we make use of a REST sender communication channel. As seen, it contains elements of various types, including an array:



A sample response message in XML format looks like:



Using standard configuration of a REST sender channel, JSON formatted response message produced from an XML formatted message given above, is:



It can be noticed that some elements’ types were interpreted incorrectly - for example:

• Element “ID” wasn't recognized as a String, but as a number - Jettison processor treated it as a number, because element value contains only numeric characters;


• Element “Properties” wasn't recognized as an array - Jettison processor treated it as a non-array object with a nested structure, because an element contains only one child entry of an element “Property” (no other sibling elements "Property").

Let’s fix this using enhanced XML/JSON conversion. In a REST sender channel, parameterization for enhanced XML/JSON conversion is done in a table “Custom XML/JSON Conversion Rules”. Below is configuration which aims troubleshooting type and conversion mismatches highlighted earlier:



After executing an interface once again and checking JSON formatted response message, it can be observed that now JSON output is produced correctly:



I didn't find details regarding parameterization in official materials, so let me summarize acceptable and valid values used in enhanced XML/JSON conversion parameters, and explanatory notes regarding their usage, in a table below:

Field	Description	Valid values
XML Namespace	XML namespace of an XML element
Prefix	XML namespace prefix of an XML element
Name	XML element name
Type	XML element type. Following types are currently supported: String, Integer, Decimal, Boolean. It makes no difference which notation for the type value is chosen as long as it is one of those mentioned in a list of valid values. If no value is specified, no specific XML/JSON conversion instructions are applied and default logic of Jettison processor is applied.	String type string xs:string xsd:string Integer type int integer xs:integer xsd:integer Decimal type decimal numeric float xs:decimal xsd:decimal Boolean type bool boolean xs:boolean xsd:boolean
Array Type	Indicator if an XML element is an array or not. It makes no difference which notation for the array type indicator value is chosen as long as it is one of those mentioned in a list of valid values. If no value is specified, array type indicator is set to "false" by default.	If element is array 1 true yes If element is not array 0 false no
Default Value	Value that will be assigned to a JSON element in case XML/JSON conversion for a corresponding XML element fails. For example, in a provided demo, value of an element “Quantity” will be processed as an integer. If original value cannot be converted to an integer (for example, it contains not only numeric characters, but its content is alpha-numeric), then JSON output will receive a default value for such an element, which is “0” in this case. It should be noted that default value is not verified against element type specified in a field "Type" - it is treated as a String. In this way, for example, it is possible to specify default value "Invalid value" for an element "Quantity" in a provided demo. An error will not be issued neither during activation of a communication channel, nor at runtime during execution of a message by REST adapter, even though provided default value mismatches with an element type (integer). Having this in mind, attention should be paid to provided default value and its compliance to an element type in sake of consistency.	Any value. Following values are treated specially: "null" (with quotation marks) - interpreted as String value "null" null (without quotation marks) - interpreted as null "" (just quotation marks) - interpreted as empty String value

Links

Please also refer to following materials discussing enhanced XML/JSON conversion in SAP REST adapter:

• SCN article PI REST Adapter – JSON to XML conversion,


• SAP Help documentation: Converting XML/JSON - Advanced Adapter Engine - SAP Library.

[engswee]

Hi Vadim,

Wow! Brilliant effort in unearthing such a hidden gem! Thanks for taking the effort to document down this feature (especially the parameter table) and sharing it!

Regards,

Eng Swee

PS: I can't view the first two images - not sure if something is missing there.

[vadimklimov]

Hi Eng Swee,

Thank you for your positive feedback! I'm glad if provided information will be helpful to others.

You are right, for some reason, first two screenshots got corrupted. I re-inserted them, they should look better now.

Regards,

Vadim

[engswee]

Hi Vadim

Thanks for updating the images. It looks fine now and gives a clearer picture of the example.

This will definitely be useful to a lot of developers. It is blogs like this that really help bridge the gaps that are sorely missing in SAP's documentation, and thus make life much easier for those of us working on PI :wink:

Kudos again for this!

Regards

Eng Swee

[harish_babu5]

Hi Vadim,

Excellent blog! I am actually facing the same error as described in this Array and object issue in JSON, REST

I am on PO 7.4 SP 12. I don't have the table “Custom XML/JSON Conversion Rules” in my REST communication channel. So I need to do a patch upgrade according to the Note. I checked the note that you have referred in this, Note 2175218.

It doesn't have a patch level for SP12. The note provides patch only for SP 11 and SP 13.

My necessity is to make a node an array even when it has only one record. So should I do a patch upgrade to 13 and then apply the patch mentioned in the Note 2175218.

Regards

Harish

[vadimklimov]

Hi Harish,

If you have a sandbox system, you may try applying most recent patches for installed SP12 and see if enhanced XML/JSON conversion functionality is contained in them and gets introduced to a system, or you may raise a request to SAP asking them if / which patches for SP12 contain this enhancement (maybe it turns out respective patch level was just missed in a list of relevant patches in a SAP Note).

Regards,

Vadim

[Former Member]

Hi Vadmin,

Are the custom conversion rules available for PI REST Receiver channel as well?

Currently our system are updated till SP11 and i dont see such options for receiver channel

thanks

Ramesh

[vadimklimov]

Hi Ramesh,

Yes, enhanced XML/JSON conversion functionality is available in both REST sender and REST receiver. If you already applied corresponding patch for SP11, which contains this feature (it shall be at least PL20 for SP11 for release 7.4), then please also ensure that adapter metadata has been updated, too (updated XI Content for software component SAP Basis has been imported in ESR). Refer to SAP Notes 1536986 and 1637955 for more details.

Regards,

Vadim

[Former Member]

Wonderful effort in unearthing behavior of JSON/XML via REST adapter .

[Muniyappan]

Thanks Vadim for sharing. I am able to use this feature after SP upgrade to 12.

is there any was to log json data in the adapter audit logs? it will be handy for troubleshooting. we can take the same payload and test it from outside PI.

Regards,

Muni

[vadimklimov]

Hi Muni,

It is possible to log JSON message in case XML/JSON conversion is used. In REST communication channels, there is a configuration option for this (checkbox "Log JSON message" - it becomes available when XML/JSON conversion is enabled), which was introduced by new feature development described in SAP Note 2175263. JSON message logging functionality was introduced earlier by new feature development described in SAP Note 2079486. When this kind of logging activated, JSON message will be logged and visible in Message Monitor, in message versions. This will not log JSON message to an audit log, but will create a message version. I'm not aware of any SAP standard functionality of REST adapter that would enable logging JSON content into message audit log. But hopefully a logged JSON message in Message Monitor will fulfil your needs.

Regards,

Vadim

[Muniyappan]

Thank you Vadim. i am able to view the json message. It is very useful for troubleshooting especially when exceptions thrown from rest service.

Regards,

Muni

[harish_babu5]

Hi Vadim,

Can you take a quick look at this thread?

http://scn.sap.com/thread/3740024

Regards

Harish

[vadimklimov]

Hello Harish,

The feature described in this blog, fulfils requirement which was raised in the referred thread. The thread is relatively old and has not been active for already a long time.

Regards,

Vadim

[Former Member]

Hi Vadim,

Is there a provision on the configuration where I can instruct the JSON to XML parser to skip certain JSON elements.

In my JSON response I am getting a comments element which contains characters which are not permitted in XML, I wanted to skip this element completely.

Below is the error I see in the REST receiver adapter:

Information Server returned code: 200

MP: exception caught with cause java.lang.RuntimeException:

com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x8) in text to output (in xml 1.1, could output as a character entity)

Exception caught by adapter framework: com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x8) in text to output (in xml 1.1, could output as a character entity)

Thx in advance

Ravijeet

[vadimklimov]

Hello Ravijeet,

The provided JSON does not look like well formed - no escaping for some characters contained in the value of the element 'comments', as well as it seems incomplete.

As for your question, I'm not aware of any functionality / parameterization in REST adapter, where you can explicitly define ignored JSON elements. Instead, when the returned JSON is well formed document, can you test its processing once again and see if implicit escaping takes place.

Regards,

Vadim

[Former Member]

Hi Vadim,

The JSON to XML parsing works good when I have no data in json element "comments". When I was doing end to end testing in DEV with no data in comments the JSON to XML thing works good. Now in UAT I see data coming in comments field and this is all raw HTML data which they capture in this comments field. Now when I get the data in "comments" field, the JSON to XML fails in the REST receiver adapter.

[vadimklimov]

Ravijeet,

First of all, this kind of long extracts of the failed message for the particular scenario are not very appropriate for blog comments - this is a use case for a new forum thread.

Regarding the error... As I wrote above, I don't think you can explicitly ignore any specific JSON element when processing the response message in REST adapter by standard means. Can you check with the called system responsibles, if they can encode symbols so that they can be parsed further?

Regards,

Vadim

[Former Member]

Hi Vadim,

Nice blog. In my scenario REST adapter is configured as receiver channel. since my case a synchronous process from ECC(Proxy)–>Request–>PI–>REST SERVICE–>Response–>PI –>ECC. Please help out with the XML/JSON conversion in the receiver channel. Looking forward for your reply. 


This is the request body which must be passed to my REST Service.

[vadimklimov]

Hello Yamini,

The message you provided as an example, corresponds to OData, but not pure generic RESTful. Hence, please check OData version used by the called service and based on that, verify if you can use OData adapter instead of REST adapter on PI side in your scenario.

Regards,
Vadim

[Former Member]

Thanks Vadim ,for the clarifications. I am new to SAP PI. As a result of my findings i could add oData adapter by installing this add-on(SAP Process Integration, connectivity add-on 1.0) to the PI 7.4 System. Can you help me out with basic links to configure a oData receiver channel with the mapping structure. 

[vadimklimov]

Yamini, please check official SAP documentation for configuration of OData receiver channels as well as materials on SAP Community such as blogs - there are already examples on how to use OData adapter in receiver flow scenarios. If after following them you will have any specific question, I would suggest you creating a separate question in Questions & Answers section - comments to the blog is not very appropriate place for having discussion on the topic that is not directly related to the subject of the blog.
Regards,
Vadim

[Former Member]

Okay got your point. Thanks Vadim.

[avinash_ayanala]

Hi Vadim,

Thanks for providing such valuable information for us.

While tring to convert the message from json to XML format using rest adapter i am not able to get the attributes into XML structure from json(attributes are declare with '@'). Is there any option available for me to get it into XML.

Thanks,
Avinash
 

[vadimklimov]

Hello Avinash,

I don't think current implementation of REST adapter supports XML attributes during JSON to XML conversion, have never seen any evidence of this.

Workaround here could be either avoidance of XML attributes in the message type and their replacement with XML elements, or usage of intermediate message type. In the latter case, intermediate message type only has XML elements and not attributes and is used when converting incoming JSON document into XML document. Afterwards, you shall be able to make use of the generated XML document (containing only XML elements) and convert it to required final target message (containing XML elements and attributes) from within the mapping - precisely, making conversion of required XML elements into their counterparts / XML attributes.

Regards,
Vadim

[emir_morillo2]

Hi,

Has this ever happened to you?

https://answers.sap.com/questions/125962/error-rest-receiver-adapter-content-is-not-allowed.html

Any help would be grateful

Regards.

[former_member233489]

Hi Vadim,

 

In my scenario, we have a REST sender where I need to convert from JSON to XML.

JSON:

{
"ReqType" : "C",
"AmendNo" : "0",
"RequestDate" : "20.01.2017",
"RequestTime" : "25",
DriverCode" : "3165",
"FirstName": "XXX",
"LastName": "XXX",

}

Converted XML:

<?xml version="1.0"?>

-<ns1:MT_Driver_C_Req xmlns:ns1="urn:ZZZZ.com:DriverMaster">

<ReqType>C</ReqType>

<AmendNo>0</AmendNo>

<RequestDate>20.01.2017</RequestDate>

<RequestTime>25</RequestTime>

<DriverCode>3165</OOMCO_DriverCode>

<FirstName>XXX</FirstName>

<LastName>XXX</LastName>

</ns1:MT_Driver_C_Req>

The problem is I have an array of driver values where I wrapped the XML with a recordset which occurs 0-unbounded. How do i get this array type in request XML? I'm using Add wrapper Element in channel to add Message Type root node in the converted XML to make mapping work.

[former_member234666]

Thank you Vadim!!! This Work Fine

 

Congratulations.

 

Regards,

[former_member211899]

Thank You Vadim for this wonderful blog. We run SAP PO 7.5 SP06 and I was working on a REST Receiver Scenario. I am unable to find the Custom XML/JSON conversion rules or the Enable Namespace mapping checkbox. Is it because we are running on a lower SP ?

I have to add the namespace details to the outgoing JSON for it to be accepted at the Receiver. I had also posted a question for this specific scenario but I guess this would be the right forum to raise my query.

[vadimklimov]

Hi Krishna,

I replied in the question thread that you raised. Summarizing that answer, please check that REST adapter metadata is refreshed (refer to SAP Note 2276231).

Regards,

Vadim

[Former Member]

Hi Vadim,

Thanks for explaining the concept, it solved my critical requirement and made me aware of one of the trickiest part of  REST receiver adapter.

Great Work I say 🙂

Regards,

Ashutosh

[former_member541603]

 

 

Hi there,

I have a problem with conversion on REST Receiver adapter, method GET. The communication channel is not converting the json payload to html, so the payload that returns to S4Hana is empty.

The interface is sycronous: SAP S4Hana (Proxy) -> SAP PO (REST) -> External System (GET method).

Here is the json returned from external system.

[{“id”:9946,”name”:”KRUPS SOLUCOES EM TI”,”enabled”:true,”chain”:{“id”:25933,”name”:”KRUPS”,”chainGroup”:{“id”:25690,”name”:”KRUPS”,”chainCode”:null}},”pointOfSaleChannel”:null,”code”:”900000000″,”companyRegistrationNumber”:”66508649299″,”companyName”:”KRUPS”,”shoppingCenter”:null,”address”:{“id”:10253,”neighborhood”:”GUARARAPES”,”address”:”RUA NOVA”,”number”:”677″,”zipCode”:”66811-320″},”region”:{“id”:14088,”name”:”KEY ACCOUNT”,”macroRegion”:{“id”:14089,”name”:”Macro Regional KEY ACCOUNT”}},”pointOfSaleType”:null,”pointOfSaleProfile”:null,”customFields”:null,”deleted”:null,”productLineMixMini”:null}]

I had configured conversion rules on communication channel, but did not work.

Bellow are the images.





 



Testing on sproxy.



I would appreciate some advices.

 

 

 

[vadimklimov]

Hello Karla,

 

When configuring custom XML/JSON conversion rules in the communication channel, please fill in namespace, prefix and name of XML element (and type / array type - where relevant) - I see that for few occurrences, they are missing.

 

Can you check trace of the communication channel for the processed request (response message), or see what XML message is generated for the response message after it gets converted from JSON to XML, and before response message mapping?

 

Regards,

Vadim

[Former Member]

Hi,

How can a proper null value be achieved in the JSON format?

I would like to put a null without quotation marks into the message.

If I simply type the null into 'Default Value' column of the "Custom XML/JSON Conversation Rules" table, the result will be

{

"Field": "null"

}

but I want to have this:

{

"Field": null

}

 

Thanks for helping.

[vadimklimov]

Hello Gergõ,

 

In REST channel configuration, when you configure data format settings, deactivate checkbox 'Treat null As String' - by default, it is activated, and any field's value 'null' in the message payload will be put in the JSON output as "null", whereas if deactivated, it will be put as null (with no quotation marks).

 

Regards,

Vadim

[surender_durgam]

Hi,

I have a problem with conversion on REST Receiver adapter, method GET. The communication channel is not converting the json payload to html, so the payload that returns to S4Hana is empty.

The interface is sycronous: SAP S4Hana (Proxy) -> SAP PO (REST) -> External System (GET method).

Here is the json returned from external system.

{"Details":{"OrderDetails":[{"OrderID":"0","OrderNo":"00010" }],"CustomerID":"00001","UserID":"","Name":"","PCode":"","Status":"Sucess","Message":"Record Inserted Successfully"}}

Here is the error message from PO system.

MP: exception caught with cause com.sap.aii.adapter.rest.ejb.parse.InvalidJSonContent: Invalid JSON message content used; Message: "JSONArray text must end with ] at character 28 of 

Bellow are the images.

Channel



 

Error Message



Please suggest me to resolve this error.

 

Regards

Surender

[Former Member]

Hi Vadim,

 

Could you please look at the below thread and suggest.

https://answers.sap.com/questions/398216/

 

 

Regards,

P

[varun_kasireddy]

Hi Team,

 

Can someone address the issue in the below link please. Appreciate your help. Its a Receiver Adapter with PUT method.

https://answers.sap.com/questions/416775/rest-receiver-adapter-xml-to-json-issue-http-error.html

 

Thanks,

Varun

[vadimklimov]

Hi Varun,

I have replied in your forum thread: please check SAP Note 2465948 and the module parameter 'setIgnoredElements' that it introduces in REST adapter to remove specific defined fields of XML message in the generated JSON message.

Regards,

Vadim

[martin_summererzwei]

Hi Vadim,

This would mean that I'm able to convert from

<n0:ANAF_Request xmlns:n0="https://s-sap.net/anaf/PlatitorTvaRest/api/v2/ws/tva">

 <RequestData>

  <cui>3234</cui>

  <data>2018-02-24</data>

 </RequestData>

 <RequestData>

  <cui>1212121</cui>

  <data>2018-02-24</data>

 </RequestData>

</n0:ANAF_Request>

 

to

[
{ "cui": 3234, "data":"2018-02-24" },
{ "cui": 1212121, "data":"2018-01-17" }
]

by

• wrapping the outer element


• setting setIgnoredElements to RequestData

Correct?

 

thanks Martin

[vadimklimov]

Hi Martin,

 

Yes, you are correct: enable 'Strip Outer Element' feature to remove root element 'ANAF_Request', and add module parameter 'setIgnoredElements' with value 'RequestData' to remove element 'RequestData' from the output.

 

Regards,

Vadim

[martin_summererzwei]

Thanks a lot!

So I start patching to have this feature available!

kr Martin

 

 

[RafaelVieira]

vadim.klimov, thanks for sharing this excellent material.

I have a question: in my PI 7.31 SP21 I have a JDBC-->REST scenario and I'm sending a URL field.

The "/" characters of this URL are being escapade and as result the URL reaches the target API as http:\/\/google.com\/flights (that's a standard behavior of Jettison processor).

I saw some other past discussions around similar issues but the solution for most of them was a Java mapping.

I'm wondering if I can have this resolved w/o Java mapping (or custom adapter modules) but rather in this Custom XML/JSON Conversion Rules section.

Would you know how to achieve this or if that's even possible?

 

Thanks,

Rafael.

[former_member59]

Hi Vadim,

Thanks for sharing this excellent article.

I am facing problem converting JSON to XML using REST adapter.

my JSON payload is like ["10xxx","19xxxx","Kerry",.....] in this payload we don't have field names as general JSON  [{ "field1": value, "field2": "value"}}].

but we are able to convert JSON to XML as attached image.

Issue is for every value wrapped into <root> tag in XML. How we can replace <root> tag with filed names. Kindly help me to resolve this issue.

Thanks in advance for all your help.

Best Regards,

[vadimklimov]

Hello Karunakar,

 

For the JSON message sample that you provided - precisely, JSON message that contains unnamed JSON array - this is fixed behaviour of REST adapter's JSON to XML conversion logic that creates XML elements for array values of unnamed JSON array, using the defaulted XML elements' name 'root', this cannot be customized in the REST communication channel.

 

Regards,

Vadim

[former_member59]

 

Hi Vadim,

 

Thanks for your response and clarification.

Best Regards,

Karunakar A

 

[former_member59]

Hi Vadim,

How to send a query to SFDC rest api using Rest adapter.

Ex :- Below query is working fine in Soap UI but when i am using in Rest adapter Fatal error is receiving in Rest adapter. Please guide me to resolve this issue.

https://***yourInstance***.salesforce.com/services/data/v20.0/query?q=SELECT+name+from+Account





Thanks in Advance for your help.



Best Regards,

[peter_wirkner]

Hi, does anyone know how to handle a sync JSON response containing "null" (as a string)? While processing our REST receiver adapter throws the error

"java.lang.NullPointerException: while trying to invoke the method java.lang.Object.toString() of a null object returned from com.sap.aii.adapter.rest.ejb.json.CustomMappedNamespaceConvention.convertType(java.lang.String, com.sap.aii.adapter.rest.ejb.json.CustomMappedNamespaceConvention$JSONType, java.lang.String)"

Seems according to RFC 7159, “null” (as well as “true” and “false”) becomes a valid JSON text

Thanks in advance

 

[former_member250735]

Hi,

This fixed the issue when the array has one element but I don't think it works if the array is empty.

I am expexting:      

"LIST": {

     "item": []

}

but i get "LIST": " "

Is there a way to make the output an array even if its empty?

 

Thanks.

 

[SushantShinde]

Hello Rafael,

Do you have any update on this other than Java mapping.

Please let me know if there's one other than Java mapping.

 

Regards,

Sushant

 

[former_member338135]

REST Adapter in PI/PO: Enhanced XML/JSON Conversion, the SAP Note 2175218 is applied in my system but not view Custom Conversion XML/JSON Rules in SAP NWDS, my SAP PO version 7.4 SP 18. can I come?