SAP Cloud Integration: move away from those IDoc messages!

[PiotrRadzki]

Introduction

I’m coming back to you with a new Blog because we need to talk about IDoc messages! 

This time I will focus more on the design and architecture of integration solutions that we as Architects and Developers create on SAP Cloud Integration, SAP PO or any other integration middleware that you use in your Organization. As SAP Integration Suite with all services included is already quite a mature iPaaS, you should already now consider migrating your legacy integrations from SAP PO (PI, XI or Business Connect if you are still there!) or any 3rd party middleware that does not fit to your Integration Strategy. At the moment of writing this blog, SAP was recognized by Gartner as a Leader in Magic Quadrant for Integration Platform as a Service with its SAP Integration Suite. I think it’s another argument to use such a leading platform for connecting to modern API’s! 

If you are already planning migration to Cloud Integration in your Team or Organization, then you can make use of multiple guidelines and tools that I listed at the end of this blog. SAP Community provides already a lot of best practices that should simplify this activity and with use of SAP inhouse or 3rd party tools and/or Consulting services, you can make this process really smooth. If you are still not sure how to design the correct architecture of your systems from the integration perspective, then let’s deep dive into Integration Strategy and define your path to the cloud or hybrid integration landscape. Proper Integration Management with modern tools is very important and can save a lot of cost and hassle - I’ve put ISA-M related references at the end of this blog as well. 

This Blog does not cover scenarios like a migration of B2B add-on interfaces on SAP PO to SAP Integration Suite Trading Partner Management, or ditching and redesigning interfaces for WM systems that work with IDoc messages. It’s more of an overview and a trigger for a discussion on how to modernize a technical landscape, starting with analyzing good old IDoc interfaces and listing options to move and implement API first strategy.

Rationale

Integration of various SAP systems with use of IDoc messages is still very popular and sometimes even the backbone of the interfaces developed between SAP ABAP stack systems and 3rd party. This is very common in Logistics or Production focused organizations. Considering migration or modernizing your interface is a great opportunity to review the technical solutions implemented some time ago and to think about potential migration to more modern and robust solutions. 

Your IDoc scenario might be complex, what looks as a simple IDoc integration might be just the tip of the iceberg. Using the opportunity of migration activities you can check and refactor the solution. Maybe you can move away from customized and extended IDoc messages and logic behind it? It might be worth checking out, maybe an alternative API provides better functionality? Maybe SAP provides an improved way to handle the process? Maybe your highly customized IDoc might be replaced with a standard or almost standard SOAP API which will have just a few extensions. Additionally, you will gain transparency and the cost of maintenance might be lower due to much more user friendly structures and field-names in the modern API's compared with IDoc message structures.

In some rare cases (but it will be happening more often), your IDoc might be even deprecated and not supported anymore. In this case, the need for migration from such a solution is a must. So, what are the alternatives and how to approach such analysis? This is especially relevant for all S/4 HANA transformations, you will face a need to or must to use the new way of doing things. It’s better to prepare yourself upfront before your IDoc becomes obsolete.

Alternatives

Below, I've listed standard ways (SAP delivered) to replace IDoc based integration's. IDoc can be also replaced with a fancy custom solution but here in this blog, I’m going to describe standard options only, so you can keep the core clean!

SOAP API

SOAP API's are great alternatives to IDoc messages. You may know them well under ABAP Proxy name as they are "hiding" behind SPROXY transaction code. On S/4 HANA as well on S/4 HANA Cloud, the list of provided SOAP API's is long and after some investigation it gives a nice overview of what can be achieved and which integration processes based on IDoc messages can be refactored with this type of API. Most of those SOAP API's go way back to SAP ERP and ECC systems, but they are being continuously upgraded. New SOAP API's are being developed and provided by SAP at present as well. . Every new version of S/4 HANA provides features and improvements for SOAP API's. This also gives us certainty about a well established solution and a code in it as well as a future proof approach of migrating IDoc based integration to SOAP API's. The advantage of the SOAP communication method against SAP S/4 HANA is that it can support asynchronous or synchronous communication, so in terms of functionality appropriate API can be found to provide the expected result based on an initial IDoc requirement. Another advantage of SOAP API's is a really good enhancement framework that gives flexibility of enhancing the ABAP Proxy structure with custom extensions and implementing encapsulated custom logic.

SOAP API's on S/4 HANA: SOAP | SAP S/4HANA | SAP API Business Hub

Enterprise Services SAP Help Page

SOAP API's on S/4 HANA Cloud: SOAP API | SAP S/4HANA Cloud | SAP API Business Hub

OData API

OData API is a flagship connectivity for S/4 HANA integration domain. For some of the SAP landscapes it might be quite new option coming up using the opportunity of S/4 HANA transformation. But a little bit more experienced integration specialist might recall SAP Gateway which was already introducing OData model and API's into SAP ecosystem. This was implemented as a separate system next to your SAP ERP/ECC box. With the current approach on S/4 HANA the whole OData framework is built into the system itself and integrated closely to the processes of S/4. With an impressive list of available CRUD (Create, Read, Update, Delete) operations on various Objects, it provides an interesting alternative to legacy integration solutions implemented so far. Of course, we need to remember that essentially the OData API's are used for synchronous communication therefore your IDoc based scenario might require deeper  analysis and redesigning effort to use OData. It's important to mention that OData API provides broader  access to the whole data model of an application/system itself that you integrate with using the expand and navigation attributes compared to SOAP or REST API's. On S/4 HANA and S/4 HANA Cloud we are getting API's that support OData V2 as well V4. V4 provides a significant improvement to the protocol itself and query methods. 

Just check below the available API's and try it out yourself!

OData V2 on S/4 HANA ODATA V2 | SAP S/4HANA | SAP API Business Hub

OData V4 on S/4 HANA ODATA V4 | SAP S/4HANA | SAP API Business Hub

OData V2 on S/4 HANA Cloud ODATA V2 | SAP S/4HANA Cloud | SAP API Business Hub

OData V4 on S/4 HANA Cloud ODATA V4 | SAP S/4HANA Cloud | SAP API Business Hub

REST API

In context of an S/4 HANA transformation the REST API's are not very popular as of now, currently there are only few REST API available for that system but in case of moving from an IDoc  based integration towards modern API, it is worth considering especially because off it's asynchronous processing mode which can be used in "fire and forget" scenarios. The library of REST API's on S/4 HANA is not very rich, however I hope SAP will go into that direction, especially for some popular asynchronous scenarios where a heavy SOAP envelope is not required and it would be a good candidate for lightweight REST service. In case of S/4 HANA Cloud the list of available REST API's is a little bit longer but it focuses mainly on API related to SAP Intelligent Trade Claims Management processes. It is worth observing this category and checking if something new comes up.

REST API's on S/4 HANA: REST | SAP S/4HANA | SAP API Business Hub

REST API's on S/4 HANA Cloud:  REST | SAP S/4HANA Cloud | SAP API Business Hub

Events

Events are quite new "objects" in the SAP ecosystem although the event itself has been known for a long time in the integration space. With S/4 HANA we are getting powerful access to Events that act as notifications and trigger for iPaaS to start various integration processes. Events in fact might be quite close in their nature to outbound IDoc messages which as we know can be triggered by change pointers and also act as event itself. With a highly promoted API-led and Event-Driven Integration approach (not without reason) the Events are an interesting approach and direction to pursue. So far Events are available for S/4 HANA mainly, we have also similar events functionality in SuccessFactors, SAP Subscription Billing or SAP Marketing Cloud as well S/4 HANA Cloud Business Events but we should expect more and more Event based integrations available in the SAP integration space.

We should not forget about the tools and services that support this Event strategy and approach. SAP Event Mesh has been available for some time to help you in orchestration and management of events between systems and applications. SAP Event Mesh acts as a centralized approach to event driven architectures and use cases. It is a fully managed cloud service that allows applications to communicate through asynchronous events - you can position it as your Event broker for publishing/consuming business events and seamless connectivity. It's a service deployed on BTP that can orchestrate SAP and non-SAP ecosystems and distribute events across your landscape.

We also have the latest product in SAP family called SAP Integration Suite Advanced Event Mesh, which is closely coupled with the iPaaS from SAP and provides an Event-driven architecture framework as a service for bigger messages and high latency. Advanced Event Mesh provides an Event Broker, a place to manage a mesh of brokers and whole Event Streaming, Event Management and Monitoring capabilities with full Event Portal features and functionality. More insights: SAP Integration Suite, Advanced Event Mesh (cloud.sap)

There is a nice Blog which describes differences, features and a little bit about the roadmap of both products: SAP Integration Suite, advanced event mesh vis-à-vis SAP Event Mesh and SAP Integration Suite. | SA...

Business Events on S/4 HANA Cloud Event Object | SAP S/4HANA Cloud Business Events | SAP API Business Hub

Events on S/4 HANA: Event Objects | SAP S/4HANA | SAP API Business Hub

Wrap up 

This is it for now! I tried to keep it short this time as this topic is so wide and deep so it's not possible to describe every and each potential scenario and alternatives. I hope we can have discussion based on that and discuss any other alternatives that you can come up with. I know I’m touching the top of iceberg here! Every SAP installation, every landscape and every integration is different and has its own critical aspects and nuances. It is definitely good to review the integration solutions that are running for years already and check what kind of new opportunity is available on the market to support business process. Highly customized IDoc are hard to maintain and cost of maintenance is raising rapidly with every new functionality or field added so my recommendation is to at least check if you can replace it with standard API's available and I hope this blog will bring some interesting findings for your specific scenario. The same approach might be applicable not only to IDoc based interfaces but to all custom APIs  (FM's, BAPI's, SOAP services, ABAP Proxies) developed in your SAP box over the years in SAP ERP/ECC systems - it's worth to take a look and review how they work and if custom code really necessary nowadays.

Please remember that there is always exception to the rule and specific IDoc based integration might be not good candidate for refactoring, there will be always some exception to the rules that you define in you Integration Strategy - and that is fine!   

Hope this Blog will be just a beginning of a journey to modern API's in your Organization and trigger necessary discussions.

What's your opinion about moving away from IDoc based process integrations? Do you see other alternatives? Let me know in comments and lets have a discussion about it!

[EDIT] I think it's also worth to add interesting point of view from michal.krawczyk2 about replacement of EDI/IDocs with API's. B2B/TPM scenarios are not directly a topic for this blog but it's worth to consider inputs in this blog as well: SAP EDI with API instead of IDOCs - is a design flaw! | LinkedIn

References 

Migration to Cloud Integration: 

Elevate from SAP Process Orchestration and Modernize your integration platform with SAP Integration ...

SAP Process Orchestration to SAP Integration Suite Migration | SAP BTP Transformation Program

Enablement guide for SAP Integration Suite Migration Assessment Capability | SAP Blogs

Migration tool in Cloud Integration Capability of SAP Integration Suite | SAP Blogs

Migration Tooling | SAP Help Portal

Integration Strategy: 

Exploring SAP’s Integration Strategy: Free eBook Available Now | SAP Blogs

SAP Integration Solution Advisory Methodology: Template version 4.0 available now | SAP Blogs

Integration Management: 

SAP Interface Management How-To Guide | Book and E-Book - by SAP PRESS (sap-press.com)

[Tomasz_Karwacki]

Piotr,

Thanks for the blog, very good overview of the potential integration ways/methods available nowadays.
For the IDoc discussion, there are still uncertainties. And not about technology that is evolving and brings new features and options that, very often, can be more reliable but about Monitoring framework and possibility for functional people to understand tools and processes behind new method. Like with SAP gateway and OData interfaces, very often you can hear comments like 'what i can use instead of IDoc monitoring transactions now ? what are these ?'. Functional area architects or consultants just want to be able to easily track the flow and see the data based on which particular operations are done in SAP from e.g. 3rd party system.

Finally, IDoc was nothing more than just Intermediary document (as name suggests) and was introduced long time back to be able to configure the system in a way that you see this document in-between the phases of inbound and outbound processing (or vice versa). With new methods, people need proper change management, as well, as new ways and tools - finally, to be able to see the data and processing status. If these are provided also for functional areas (not only tech teams close to the integration), this should be more smooth, the transition itself.

[Jelena]

Correct spelling is IDoc. 🙂

Michal's article talks about it and Tomasz's comment too. The value of IDocs is not in just performing the main business function (e.g. post a sales order or update material master) but in all the nice additional functionality that has been built around them over many years.

There are transactions to see what's inside each IDoc and you can update information and easily reprocess it. Try that with a Gateway service, much less some web API. There are transactions to generate IDocs, post, re-post, view errors, search for data inside the IDocs, etc. It's a lot of functionality that for other options does not exist or is all over the place or just sucks. (I like Gateway a lot but error monitoring transactions there are rather clunky.)

As Tomasz correctly pointed out, IDocs are somewhat business friendly. I had no problem teaching sales reps how to use WE05 or WE19 because there is a clear structure to data. Will a sales rep be able to easily monitor an API? Without additional development, I don't think so.

All these items are easy to overlook. If SAP is serious about getting rid of IDocs, they need to get it together and provide better monitoring and reprocessing tools that non-IT personnel can also use.

[PiotrRadzki]

Hi Jelena, Thanks for your valuable opinion on this! Just short answer at the beginning - you are completely right about spelling, I corrected the Blog content and title, and fixed the spelling to the correct one.

I will come back to your opinion and comment shortly!

BR, Piotr

[dr_ajt]

Have to agree with you on this.

I've worked with BAPIs coming in from an external system and IDocs. The beauty of an IDoc is that there is a foot print left behind and when a human has to fix the message there is something to see and an audit trail when the data has been corrected.

When a BAPI fails, and by extension a REST message, SOAP message, OData call or any thing else, the problem bounces back to the source system and there is nothing for the user in the SAP system to do until the message returns back corrected.

In my experience error handling is complex and tedious to build and often doesn't happen, so the IDoc leaves the SAP system with a visible error that a human can fix within the SAP standard system.

I'm all for fancy new API calls, but unless there is a way for humans to work with them using standard tools, businesses wont accept those solutions.

 

[Jelena]

Thanks for doing this! Really appreciate your attention to detail. Looking forward to further comments.

[rhviana]

Congrats for the blog Piort !!! Keep posting, welcome !!!

[Jelena]

Link to the corresponding LI post which has many additional comments that the readers might also be interested in.

[PiotrRadzki]

Thank you for adding it.

BR, Piotr

[PiotrRadzki]

Thanks Viana, your blogs are actually great example how to nicely explain the technical scenarios. Hope you also keep on Blogging! See you out there on the projects 🙂

BR, Piotr

[Ryan-Crosby]

I don't see them ever going away unless there is a substantial framework (a free one unlike AIF) put in place to allow all those functions you have noted above in conjunction with said APIs... that said I doubt it would ever happen - simply an investment that isn't worth it because there is no sense in "fixing" what isn't broken.  I keep hearing speculation about IDocs and how they will become obsolete and SAP is moving away from them... take a peek at the Integration Advisor MAG content and what you see will suggest otherwise.  I'll pose another option... API and IDoc can coexist within the SAP landscape as they each have their uses.

IDoc is merely an internal SAP representation of EDIFACT which is quite obvious when one looks at names like ORDERS, ORDRSP, DESADV, DELFOR, FINSTA and realizes that they exist within the EDIFACT standard.  I suspect that SAP realized they were useful in other spaces which is why ALE was birthed.  If I receive an asynchronous EDI message that I need to post to the system with business data, then I'm not going to bother finding an API and deal with what goes wrong when I only need to post an IDoc and let the underlying workflow framework take care of creating an inbox item for the appropriate business users to review.  When a failure happens with IDoc everything is saved including status updates with every phase of processing, but with a lot of APIs it is left to the client to determine how proceed.  That leaves a gaping hole that can only be filled in the middleware, but I only have 10 fingers and maybe a piece of gum.

[Jelena]

You got that right. That's why multiple experts have been saying that in EDI scenarios, IDocs work just fine (that's exactly what they are meant for) and replacing them with APIs would be a loss. They can be replaced by APIs in the non-EDI interfaces and especially in scenarios where IDocs shouldn't have been used in the first place.

[fernando_siqueira]

Hi Jelena,

Do you think the same about Events (ex.:ASAPIO)? I keep hearing it could be an interesting alternative, but not sure to what extent. Do you have any experience you could share?

Best regards.

[Jelena]

It's important to understand that IDoc is a data container. It has the structure that conforms to EDI standards, which makes them perfectly suitable for such interfaces. By themselves, the IDocs don't do anything but, as noted here, over time (really long time!) they were outfitted with some nice monitoring, post-processing, etc. functionality.

The IDocs could probably be used with the event-based architecture but these are not interchangeable concepts. IDocs carry information and events do things.

Unrelated to that, I think event-based architecture in general is a good thing and I'm glad to see SAP working on it. It's just the question whether they'll carry it through and deliver great results and whether everyone will have to pay extra to use it. I wrote about it here a year ago but haven't check for updates.

[balaji_j]

Hi piotr_radzki ,

Thank you for this round up blog! Finally we can move away from doing a graphical mapping for Inbound IDoc header fields in every CPI environment.

Thanks,

Bala.

[PiotrRadzki]

I'm happy that you enjoyed!

[pazabel]

Dear Piotr,

you are pointing out "Considering migration or modernizing your interface is a great opportunity to review the technical solutions implemented some time ago and to think about potential migration to more modern and robust solutions. ".

I do not think, that wou find any more robust system than IDoc. Idoc is there for decades. In 20 years of SAP ERP integration experience I have never had any reason to loose trust in SAP Idoc infrasxtructure. I do not know any part of SAP which I would consider any more stable than Idoc.

The only reason I would consider Idoc as not being in a stable running state is, when the interface infrastructure was not installed correctly and either authorisations, Jobs, events or workflows haven't been configured.

Additionally Idoc is often missunderstood as being an alternative to SOAP. IDoc is adressing a lot of more layers of ISO OSI.

Idoc can be transferred by file plain/xml, by HTTP/XML, RFC, and other transfer methods.

Idoc is giving a good standard digital representation of business objects by either being BAPI IDocs or UNEDIFACT similar "old" IDocs like ORDERS; INVOIC and others.

Idoc is providing envelope, data and processing log in one view.

You can search for specific business content (e.g. I would like to search for all messages for debtor 12334); as of my knowledge SOAP messages are dropped afer 30 days and cannot be searched regarding specific business content. Within ITIL RUN this is a key question an an interface issue.

IDocs are providing queuing, so ensure that changes are done in the correct sequence.

IDocs can be linked to business objects (eg. Outboudn invoice from SD to IDoc to inbound MM invoice).

Idoc monitoring is available out of the box BD87, WLF_IDOC and can be used by the business. SRT_MONI and /AIF/MON are more complicated and not likely to be used by the business.

It is very easy to configure error or success workflows to inform the business on any issue.

Error idocs/ messages can be grouped by type of error.

Changes in IDoc data is possible and is well documented by copying the original Idoc and logging the changing user and datetime.

And I think I will have missed a lot other points here.

I hope SAP will understand that it is not making any sense to drop Idoctechnology.

 

Best regards Patrick 

 

[helmut_skolaut3]

The problematic is always who is responsible for the data quality of this interface. I love the idea to push this back to the sender application and let them fix all the problems that could occur - then synchronous APIs, BAPIs, SOAP Services and all such technologies are perfect.

As soon as the data quality can only be worked on in the receiving system we need some kind of cockpits where someone can watch the not processed data and either correct the payload or maybe correct the missing master data in the system. Currently i haven't found anything like this is (yet) available - at least not for APIs.

User interfaces like AIF are not user friendly enough that a business user can work with to mitigate the problems - i don't know if they can be used for APIs?

BR Helmut