Introduction
This is officially part II of the series that started with SAP EDI/IDoc Communication in CPI Using Bundling and Fewer Mapping Artifacts, which will deal with handling incoming EDI communication, and reuse capabilities of the outgoing communication flow shown in the first part to send functional acknowledgements.
Notes
Archiving was not addressed because we do not have a CMS system that can be used with the latest features that have been made available. The archiving function is also indiscriminate and does not offer functionality to peek at the contents to determine if archiving is deemed necessary. Any custom archiving function that evaluates contents would execute for every message split iteration, even if it only makes sense to save one archive once at the beginning of message processing. At this time a Gather step was also not evaluated to package IDocs, but there are plans to do so in the future when we implement something aside from incoming 997s. One very important note about handling a Gather step is that it would be advisable to not accept messages with multiple functional groups in a single interchange - e.g. 944s and 945s at the same time which would map to WMMBXY and SHPCON respectively, and have the IDoc XML co-mingled after the Gather step. (Gather has been evaluated and have opted to not use it for back-end IDoc posting) This follows the same implementation strategy of the outgoing messages from part I of the series (The only difference between the two being that outgoing messages offer extended post-processing and incoming messages offer extended pre-processing).
* Revision 1.1 - add mapping step for IDoc control record assembly with XSLT mapping - 2022-07-14
* Revision 1.2 - add additional error check for agreement that has not been maintained - 2023-01-13
* Revision 1.3 – add support for 997s with acceptance/rejection status only at interchange level (AK9) – 2023-01-26
* Revision 1.4 – Change datastore read operation to read group control number from AK102 for matching MUST set group control number to same value as interchange on outgoing transmissions in envelope – 2023-02-02
* Revision 1.5 – Change incoming 997 mapping to record REFINT information from same AK102 value NOTE we are only interested in recording our outgoing interchange for support because of our limited archiving of 810s and possibly 997s (arbitrary sender interchanges are of no assistance) – 2023-02-03
* Revision 1.6 – add process step to allow for archiving of outgoing 997 acknowledgements – 2023-02-06
* Revision 1.7 – remove IDoc control assembly process step because it can be handled with global parameters in the Integration Advisor MAG – 2023-02-09
* Revision 1.8 – remove email alerting in favor of Alert Notification Service (see part I for referenced blog on ANS setup) – 2023-02-17
Revision 1.9 – add Github link for package download – 2023-10-28
Transformation Flow
The transformation flow consists of two main branches:
- The functional acknowledgement handling based on the envelope inspection and validation handling of the EDI splitter.
- The core EDI transformation flow that is split further into two types of processing - i.e. recording 997s as SYSTAT01 IDocs or converting agreed upon EDI messages into the necessary IDocs for posting.
The integration flow includes an exception subprocess to trigger a secondary flow on error (that ends in error), so the message will remain in retry mode in the JMS queue. The error generated in the secondary flow is configured for tracking in the Alert Notification Service.
Initial Processing and EDI Splitter
The initial content modifier step allows setup of basic information for IDoc posting - e.g. Logical system, client, receiver partner name, etc. Not a whole lot to say about the EDI splitter step other than its use is well documented here - Define EDI Splitter.
Functional acknowledgement vs. EDI Handling
The functional acknowledgement branch handling is determined by the following evaluation of the splitter outcome.
There are two additional process steps in the functional acknowledgement branch - the first step is a groovy script to evaluate the archiving option for outgoing 997s and remove the acknowledgement flag (Some systems will return a TA that isn't supported here), and a second content modifier step to pass the value from the SenderPid header to the pid header, which is used in the outgoing communication flow from SAP EDI/IDoc Communication in CPI Using Bundling and Fewer Mapping Artifacts. Otherwise, the default route passes the split message to the core EDI branch for processing. Below is the groovy script to evaluate the archive settings from the partner directory and a sample json file for activating archiving.
import com.sap.gateway.ip.core.customdev.util.Message
import java.util.HashMap
import com.sap.it.api.pd.PartnerDirectoryService
import com.sap.it.api.ITApiFactory
import groovy.json.*
def Message processData(Message message) {
def service = ITApiFactory.getApi(PartnerDirectoryService.class, null)
if (service == null){
throw new IllegalStateException("Partner Directory Service not found")
}
// Read 997 parameters if they exist to determine if archiving should be performed
def headers = message.getHeaders()
def SenderPid = headers.get("SenderPid")
def ReceiverPid = headers.get("ReceiverPid")
def slurper = new JsonSlurper()
def isArchivingActive = service.getParameter("IsArchivingActive", ReceiverPid, java.lang.String.class)
if(isArchivingActive == "true") {
def msgParamsBinary = service.getParameter("997", ReceiverPid, com.sap.it.api.pd.BinaryData)
if(msgParamsBinary != null) {
def msgParameters = slurper.parse(msgParamsBinary.getData(), "UTF-8")
if(msgParameters.Outbound?.ArchiveMessage == "true") {
message.setHeader("ArchiveMessage", msgParameters.Outbound.ArchiveMessage)
def folder = headers.get("ArchiveFolder")
message.setHeader("ArchiveFolder", folder + SenderPid + "/")
} else {
message.setHeader("ArchiveFolder", null)
}
}
}
// Remove acknowledgement flag
def body = message.getBody(java.lang.String.class)
def newBody = body.substring(0, 100) + "0" + body.substring(101)
message.setBody(newBody)
return message
}{
"Outbound": {
"ArchiveMessage": "true"
},
"Inbound": {}
}Lookup Partner Directory Information and Execute Basic Partner Validation
After gathering some key headers from the message the information is evaluated first to determine if a message has been received by an unknown partner, unknown sender, or if an agreement has not been maintained. In any of those cases the message is flagged in error status, and the payload is saved as an attachment, a message log entry is added, and the message body is adjusted to contain an alert. The message will then be directed in the next steps of the flow to trigger an email to the interested parties, and have the message end in escalation status such that it can be reviewed - this is also done to avoid endless JMS retries. The router step for escalation status in the Partner Error Step is the following logical statement - ${property.UnknownSender} = 'true' or ${property.UnknownReceiver} = 'true' or ${property.NoAgreement} = 'true'. If basic message validation is successful then core message data is read to determine the IDoc target (and anything we choose to add later). The last steps of the script get IDoc partner information for the sender and all relevant mapping XSL transforms along with partner agreement data that will reflect if extended pre processing should be executed on the message. The 997 messages differ slightly in processing where IDoc numbers are retrieved from a datastore and mappings are not called from the partner directory, hence the exclusion in the associated if block.
import com.sap.gateway.ip.core.customdev.util.Message
import java.util.HashMap
import com.sap.it.api.pd.PartnerDirectoryService
import com.sap.it.api.ITApiFactory
import groovy.json.*
def Message processData(Message message) {
def service = ITApiFactory.getApi(PartnerDirectoryService.class, null)
if (service == null){
throw new IllegalStateException("Partner Directory Service not found")
}
// Read partner data, and EDI message information for conversion and
// communication purposes
def headers = message.getHeaders()
def SenderPid = headers.get("SenderPid")
def ReceiverPid = headers.get("ReceiverPid")
def std = headers.get("SAP_EDI_Document_Standard")
def stdmes = headers.get("SAP_EDI_Message_Type")
def stdvrs = headers.get("SAP_EDI_Message_Version")
def sndql = headers.get("SAP_EDI_Sender_ID_Qualifier")
def sndid = headers.get("SAP_EDI_Sender_ID")
def rcvql = headers.get("SAP_EDI_Receiver_ID_Qualifier")
def rcvid = headers.get("SAP_EDI_Receiver_ID")
def intchg = headers.get("SAP_EDI_Interchange_Control_Number")
// Get their id and qualifier, and our id and qualifier for matching and
// validation
def x12qualifier = service.getParameter("x12_qualifier", SenderPid, java.lang.String.class)
def x12id = service.getParameter("x12_id", SenderPid, java.lang.String.class)
def messageLog = messageLogFactory.getMessageLog(message)
def partnerError = false
if(x12qualifier != sndql || x12id != sndid.trim()) {
message.setProperty("UnknownSender", "true")
messageLog = messageLogFactory.getMessageLog(message)
messageLog.setStringProperty("Unknown Sender", "Sender ID - " + sndql + ":" + sndid.trim() + " not registered for partner " + SenderPid)
partnerError = true
}
def our_x12qualifier = service.getParameter("x12_qualifier", ReceiverPid, java.lang.String.class)
def our_x12id = service.getParameter("x12_id", ReceiverPid, java.lang.String.class)
if(our_x12qualifier != rcvql || our_x12id != rcvid.trim()) {
message.setProperty("UnknownReceiver", "true")
messageLog.setStringProperty("Unknown Receiver", "Receiver ID - " + rcvql + ":" + rcvid.trim() + " not registered")
partnerError = true
}
// Read extra parameters for message handling - e.g. message target, etc.
def slurper = new JsonSlurper()
def target
def msgParamsBinary = service.getParameter(stdmes, ReceiverPid, com.sap.it.api.pd.BinaryData)
if(msgParamsBinary != null) {
def msgParameters = slurper.parse(msgParamsBinary.getData(), "UTF-8")
target = msgParameters.Inbound.Target
}
// Setup header variables from PD for conversion handling (AT items)
def alternativePartnerId = service.getAlternativePartnerId("SAP SE", "IDOC", SenderPid)
message.setHeader("SenderPartner", alternativePartnerId)
def partnerType = service.getParameter("PartnerType", SenderPid, java.lang.String.class)
message.setHeader("SenderPartnerType", partnerType)
def msgInfo = std + "_" + stdmes + "_" + stdvrs
message.setHeader("SND_CONVERSION_XSD", "pd:" + ReceiverPid + ":" + msgInfo + ":Binary")
if(stdmes != "997") {
message.setHeader("PREPROC_XSLT", "pd:" + ReceiverPid + ":" + msgInfo + "_preproc:Binary")
message.setHeader("MAPPING_XSLT", "pd:" + ReceiverPid + ":" + msgInfo + "_to_" + target + ":Binary")
message.setHeader("POSTPROC_XSLT", "pd:" + ReceiverPid + ":" + target + "_postproc:Binary")
// Retrieve partner specific information regarding converters and agreement
// information for extended pre processing
def agreementParamsBinary = service.getParameter("Agreements", SenderPid, com.sap.it.api.pd.BinaryData)
if(agreementParamsBinary != null) {
def agreementParameters = slurper.parse(agreementParamsBinary.getData(), "UTF-8")
def msgAgreement = agreementParameters.Agreements.Inbound.find{ it.Message == msgInfo }
if(msgAgreement == null) {
message.setProperty("NoAgreement", "true")
messageLog = messageLogFactory.getMessageLog(message)
messageLog.setStringProperty("No Agreement", "No agreement maintained for partner " + SenderPid + " for " + msgInfo)
partnerError = true
}
if(msgAgreement?.DoExtendedPreProcessing == "true") {
message.setProperty("DoExtPreprocessing", msgAgreement.DoExtendedPreProcessing)
message.setHeader("EXT_PREPROC_XSLT", "pd:" + SenderPid + ":ext_" + msgInfo + "_preproc:Binary")
}
} else {
message.setProperty("NoAgreement", "true")
messageLog = messageLogFactory.getMessageLog(message)
messageLog.setStringProperty("No Agreement", "No agreement maintained for partner " + SenderPid + " for " + msgInfo)
partnerError = true
}
}
// Only need to log payload once in case both errors are triggered and set body for alert
// to check escalation logs
if(partnerError) {
def body_bytes = message.getBody(byte[].class)
messageLog.addAttachmentAsString("Payload", new String(body_bytes, "UTF-8"), "text/plain")
message.setBody("Unknown sender, receiver or agreement does not exist, please check escalated messages for details")
}
return message
}Map 997
The map 997 local integration process has two core steps - read IDoc numbers from the data store (with delete) and map the incoming 997 accepted/rejected status into the corresponding 16/17 status for the associated IDocs. Below is a screenshot of the configuration of the datastore operation and the XSL transform to generate the STATUS.SYSTAT01 IDocs. Make note that the T/P indicator for the test/productive is passed to the IDoc control record along with the interchange, group and transaction control numbers for correlation purposes. An exception process is added with a message end result to ensure that should the data store read fail that the message will not go into a repetitive JMS retry loop. This outcome would result in some outgoing IDocs that cannot be correlated to an acknowledgement because the interchange no longer exists in the data store (hopefully this should never happen).
Background on 1.4 update - the SAP implementation seemingly returns the same interchange number on outgoing 997s, but it appears that most, if not all other EDI platforms will return an interchange number that is based on a sender number range and will not match. The result is that the datastore read will fail or read a wrong interchange in the unlikely event of a match to another interchange. The guidance and documentation suggests using other segments to correlate to the original interchange, but I suspect this is why I have observed in previous experience that folks set the group control number to be the same as the interchange number. It never made sense previously because the only requirement is that the group must be unique within the interchange, but without using the interchange for the group too, then how do we manage to match acknowledgements without additional rigging. Anyhow, the 1.4 changes consisted passing the interchange number as the group control number, and changing the datastore operation to read based on the value in AK102.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:multimap="http://sap.com/xi/XI/SplitAndMerge" xmlns:hci="http://sap.com/it/" exclude-result-prefixes="hci">
<xsl:param name="Client"/>
<xsl:param name="SAP_ISA_Usage_Indicator"/>
<xsl:param name="SAP_EDI_Message_Version"/>
<xsl:param name="SenderPort"/>
<xsl:param name="SenderPartnerType"/>
<xsl:param name="SenderPartner"/>
<xsl:param name="LogicalSystem"/>
<xsl:param name="SAP_EDI_Interchange_Control_Number"/>
<xsl:param name="SAP_EDI_GS_Control_Number"/>
<xsl:param name="SAP_ST_Control_Number"/>
<xsl:template match="/">
<SYSTAT01>
<IDOC BEGIN="1">
<EDI_DC40 SEGMENT="1">
<TABNAM>EDI_DC40</TABNAM>
<MANDT><xsl:value-of select="$Client"/></MANDT>
<DIRECT>2</DIRECT>
<TEST>
<xsl:choose>
<xsl:when test="$SAP_ISA_Usage_Indicator = 'T'">X</xsl:when>
<xsl:otherwise/>
</xsl:choose>
</TEST>
<IDOCTYP>SYSTAT01</IDOCTYP>
<MESTYP>STATUS</MESTYP>
<STD>X</STD>
<STDVRS><xsl:value-of select="$SAP_EDI_Message_Version"/></STDVRS>
<STDMES>997</STDMES>
<SNDPOR><xsl:value-of select="$SenderPort"/></SNDPOR>
<SNDPRT><xsl:value-of select="$SenderPartnerType"/></SNDPRT>
<SNDPRN><xsl:value-of select="$SenderPartner"/></SNDPRN>
<RCVPRT>LS</RCVPRT>
<RCVPRN><xsl:value-of select="$LogicalSystem"/></RCVPRN>
<REFINT><xsl:value-of select="$SAP_EDI_Interchange_Control_Number"/></REFINT>
<REFGRP><xsl:value-of select="$SAP_EDI_GS_Control_Number"/></REFGRP>
<REFMES><xsl:value-of select="$SAP_ST_Control_Number"/></REFMES>
</EDI_DC40>
<xsl:choose>
<xsl:when test="exists(*/multimap:Message1/*/*/G_AK2)">
<xsl:for-each select="*/multimap:Message1/*/*/G_AK2">
<xsl:variable name="position" select="number(S_AK2/D_329)"/>
<E1STATS SEGMENT="1">
<TABNAM>EDI_DS</TABNAM>
<DOCNUM><xsl:value-of select="../../../../multimap:Message2/Interchange/DOCNUM[$position]"/></DOCNUM>
<STATUS>
<xsl:choose>
<xsl:when test="S_AK5/D_717 = 'A'">16</xsl:when>
<xsl:otherwise>17</xsl:otherwise>
</xsl:choose>
</STATUS>
<REFINT><xsl:value-of select="../S_AK1/D_28"/></REFINT>
<REFGRP><xsl:value-of select="../S_AK1/D_28"/></REFGRP>
<REFMES><xsl:value-of select="S_AK2/D_329"/></REFMES>
</E1STATS>
</xsl:for-each>
</xsl:when>
<xsl:otherwise>
<xsl:for-each select="*/multimap:Message2/Interchange/DOCNUM">
<E1STATS SEGMENT="1">
<TABNAM>EDI_DS</TABNAM>
<DOCNUM><xsl:value-of select="."/></DOCNUM>
<STATUS>
<xsl:choose>
<xsl:when test="../../../multimap:Message1/*/*/S_AK9/D_715 = 'A'">16</xsl:when>
<xsl:otherwise>17</xsl:otherwise>
</xsl:choose>
</STATUS>
<REFINT><xsl:value-of select="../../../multimap:Message1/*/*/S_AK1/D_28"/></REFINT>
<REFGRP><xsl:value-of select="../../../multimap:Message1/*/*/S_AK1/D_28"/></REFGRP>
<REFMES><xsl:value-of select="format-number(position(), '000000000')"/></REFMES>
</E1STATS>
</xsl:for-each>
</xsl:otherwise>
</xsl:choose>
</IDOC>
</SYSTAT01>
</xsl:template>
</xsl:stylesheet>Map Interchange
The map interchange local integration process consists of the four core steps (extended pre is optional) - extended X12 pre-processing, X12 pre-processing, mapping, and IDoc post-processing. The extended pre-processing artifacts exist in the sender assigned space of the partner directory, whilst the remaining objects are centralized in our own designated space. Please see the picture reference below for an example of global parameters in the Integration Advisor MAG implementation - the source and target mapping should be self explanatory based on examination of the IDoc control record.
The Last Step
The final content modifier is there to set a header for pid which is used in the communication flow, but it differs from the value set in the functional acknowledgement branch because in this case it will be assigned the value from ReceiverPid. The use of a single header parameter pid with the additional content modifier steps in this integration flow allowed for the reuse of the existing outgoing communication flow created in part I for partner communication and IDoc communication to our systems. Yay for reuse and less artifacts!
Conclusion
Some core functionalities for EDI processing have not been explored yet (gather for incoming IDoc bundling noted above), so I will probably be undertaking a part III in the future. However, this build out enables the core communication flow into an SAP environment and felt it was worthwhile to share the experience with other integration techies. I encourage folks to comment, like, and ask any questions. Cheers!
References
https://help.sap.com/docs/CLOUD_INTEGRATION/987273656c2f47d2aca4e0bfce26c594/584a3beb81454d40acb052a...
https://github.com/ryanc35/X12EDIinCPI
- SAP Managed Tags:
6 Comments
