With the 2.38 version of SAP Cloud Integration an improved representation of the message processing log with embedded integration flow tracing capabilities has been released. This blog gives an overview of the new feature and provides best practices and examples on how to troubleshoot errors occurring during message processing.
Troubleshooting errors occurring during message processing is one of the most crucial tasks for an integration developer. SAP Cloud Integration consolidates all capabilities supporting the user in analyzing such error situations in the "Monitor Message Processing" feature in the Web application of SAP Cloud Integration.
With the 2.38 version of SAP Cloud Integration the "Monitor Message Processing" feature has been extended as follows:
In the following it is outlined how the various capabilities can be used in the context of a typical troubleshooting use case.
The starting point for investigating on an error occurred during integration flow execution is the "Monitor Message Processing" feature. You can use various search options to retrieve the message you are interested in.
When clicking on a message all attached log data such as processing status, log attachments, etc. are displayed in the detailed section of the page. As part of the "Status" section you can find the error message which has been thrown during integration flow execution. In many cases this error message already gives a good indication on what caused the integration flow execution to fail. For example, the error message exposed in the following screenshot indicates that an error occurred during execution of an Apache Groovy script.
In case the error message does not indicate on how to resolve the error situation you have the option to continue your investigation by looking into the message processing log. Today's message processing log is relying on the technical logging of the underlying technologies. In case you are familiar with Apache Camel, Apache CXF, etc. it is a very powerful tool to understand what exactly happened during an integration flow execution.
However, SAP acknowledge that understanding such a technical logging requires expertise in the used technologies such as Apache Camel, Apache CXF, etc. Since not all users are familiar with these technologies it is planned to continuously extend the logging by a more explicit logging per processing step. With that going forward most use cases will not require the user to look into the technical logging provided by the underlying technologies. The technical logging will only serve as backup for experts relying in their work on the technical details.
As part of the 2.38 version of SAP Cloud Integration the visualization of the log data has been evolved. The actual message processing log is displayed in a table format. Each line in the table represents a technical processing step as captured by the underlying technologies. The user can easily locate the step where the error occurred via the
icon. Clicking on the icon opens the error message in a message box.
The log data is correlated with the integration flow model. When selecting a step in the log table the correlated flow-step in the integration flow model is highlighted. This is a powerful tool since it clearly outlines the flow-step in the integration flow model a certain log entry belongs to. For example, it allows to locate the flow-step where the actual error occurred by simply selecting the step flagged with the icon. 
In regards to the troubleshooting use case outlined in this blog it might be required to validate the actual integration flow configuration since a wrong configuration can be the cause for an error during integration flow execution. You can access the integration flow configuration of the selected step in the log table by navigating to the step details using the
icon. To actually change the integration flow configuration you need to switch to the Designer.
Although the message processing log provides detailed logging for an integration flow execution insights into the content of the processed message might be required for certain cases as well. For example, in case the message processing relies on the message content (e.g. mapping, router) it is required to trace the content of the processed message along the integration flow execution.
With the 2.38 version of SAP Cloud Integration such a tracing capability has been introduced in the "Monitor Message Processing" feature. To activate the capability you need to switch to log level "Trace" for the respective integration flow. You can set the log level in the "Manage Integration Content" feature. To generate the trace data in SAP Cloud Integration you need to trigger a reprocessing of the message (e.g. by sending the message again to the SAP Cloud Integration tenant).
Back in the log view of the "Monitor Message Processing" feature you can continue your troubleshooting work. One of the capabilities offered with log level "Trace" is a visualization of the execution path of the message in the integration flow model. This so-called message traversal path is visualized by decorating the integration flow model with the
icon. A red colored icon (
) indicates the flow-step where an error was thrown during integration flow execution.
Today the message traversal path can only be visualized for log level "Trace". Going forward it is planned to visualize the message traversal data in the integration flow model for the other log levels as well.
To access the content of the processed message you can navigate to the trace data written for a selected step using theicon. In the "Message Content (Before)" tab both the message header and payload are displayed with its state before the message was processed by the respective step.
With the 2.38 version of SAP Cloud Integration it is possible to trace the message content along the integration flow execution path. Trace data are written for the processing steps modeled in the integration flow. For adapters used in the integration flow the trace data are not yet exposed in the Web application. In case you are interested in the trace data written by adapters you can download it using the Eclipse Integration Operations feature. Going forward it is planned to support tracing for adapters in the "Monitor Integration Content" feature of the Web application as well.
Finally, in the context of troubleshooting error situations occurring during integration flow execution it is worth to outline two additional rather specific use cases:
(1) In certain situation the message processing runs into an error at a quite early step during inbound processing. At this point in time no message processing log is available. To troubleshoot such error situations you can use the system logs offered for download in the "Monitor System Logs" feature.
(2) In case your integration flow runs into a retry cycle the individual runs of each integration flow execution are displayed in the "Logs" section of the "Monitor Message Processing" feature. Up to 100 runs are listed in the table. You can access the message processing log of each individual run using the hyperlink offered in the "Log Level" column of the table.
I hope you will enjoy using the new capabilities. In case of questions or feedback please feel free to comment on this blog.
References:
Cloud Integration – Setting the Log Level for Message Processing
Cloud Integration - System Log Download
Cloud Integration - Enabling Trace for Message Processing
Avoid Storing Payloads in the Message Processing Log, Especially in Productive Integration Flows
How to Avoid Excessive Storage Load Caused by Using MPL Attachments for Message Logging
Cloud Integration - Apache Camel Exchange Properties recorded as Trace Data
Troubleshooting errors occurring during message processing is one of the most crucial tasks for an integration developer. SAP Cloud Integration consolidates all capabilities supporting the user in analyzing such error situations in the "Monitor Message Processing" feature in the Web application of SAP Cloud Integration.
With the 2.38 version of SAP Cloud Integration the "Monitor Message Processing" feature has been extended as follows:
- Improved log visualization.
- Log data correlated with integration flow model.
- Embedded integration flow tracing.
- Access to integration flow configuration.
- Better insights into retry scenarios.
- Log and payload data downloadable.
In the following it is outlined how the various capabilities can be used in the context of a typical troubleshooting use case.
The starting point for investigating on an error occurred during integration flow execution is the "Monitor Message Processing" feature. You can use various search options to retrieve the message you are interested in.
When clicking on a message all attached log data such as processing status, log attachments, etc. are displayed in the detailed section of the page. As part of the "Status" section you can find the error message which has been thrown during integration flow execution. In many cases this error message already gives a good indication on what caused the integration flow execution to fail. For example, the error message exposed in the following screenshot indicates that an error occurred during execution of an Apache Groovy script.
In case the error message does not indicate on how to resolve the error situation you have the option to continue your investigation by looking into the message processing log. Today's message processing log is relying on the technical logging of the underlying technologies. In case you are familiar with Apache Camel, Apache CXF, etc. it is a very powerful tool to understand what exactly happened during an integration flow execution.
However, SAP acknowledge that understanding such a technical logging requires expertise in the used technologies such as Apache Camel, Apache CXF, etc. Since not all users are familiar with these technologies it is planned to continuously extend the logging by a more explicit logging per processing step. With that going forward most use cases will not require the user to look into the technical logging provided by the underlying technologies. The technical logging will only serve as backup for experts relying in their work on the technical details.
As part of the 2.38 version of SAP Cloud Integration the visualization of the log data has been evolved. The actual message processing log is displayed in a table format. Each line in the table represents a technical processing step as captured by the underlying technologies. The user can easily locate the step where the error occurred via the
icon. Clicking on the icon opens the error message in a message box.The log data is correlated with the integration flow model. When selecting a step in the log table the correlated flow-step in the integration flow model is highlighted. This is a powerful tool since it clearly outlines the flow-step in the integration flow model a certain log entry belongs to. For example, it allows to locate the flow-step where the actual error occurred by simply selecting the step flagged with the
icon. In regards to the troubleshooting use case outlined in this blog it might be required to validate the actual integration flow configuration since a wrong configuration can be the cause for an error during integration flow execution. You can access the integration flow configuration of the selected step in the log table by navigating to the step details using the
icon. To actually change the integration flow configuration you need to switch to the Designer.Although the message processing log provides detailed logging for an integration flow execution insights into the content of the processed message might be required for certain cases as well. For example, in case the message processing relies on the message content (e.g. mapping, router) it is required to trace the content of the processed message along the integration flow execution.
With the 2.38 version of SAP Cloud Integration such a tracing capability has been introduced in the "Monitor Message Processing" feature. To activate the capability you need to switch to log level "Trace" for the respective integration flow. You can set the log level in the "Manage Integration Content" feature. To generate the trace data in SAP Cloud Integration you need to trigger a reprocessing of the message (e.g. by sending the message again to the SAP Cloud Integration tenant).
Back in the log view of the "Monitor Message Processing" feature you can continue your troubleshooting work. One of the capabilities offered with log level "Trace" is a visualization of the execution path of the message in the integration flow model. This so-called message traversal path is visualized by decorating the integration flow model with the
icon. A red colored icon () indicates the flow-step where an error was thrown during integration flow execution.Today the message traversal path can only be visualized for log level "Trace". Going forward it is planned to visualize the message traversal data in the integration flow model for the other log levels as well.
To access the content of the processed message you can navigate to the trace data written for a selected step using the
icon. In the "Message Content (Before)" tab both the message header and payload are displayed with its state before the message was processed by the respective step.With the 2.38 version of SAP Cloud Integration it is possible to trace the message content along the integration flow execution path. Trace data are written for the processing steps modeled in the integration flow. For adapters used in the integration flow the trace data are not yet exposed in the Web application. In case you are interested in the trace data written by adapters you can download it using the Eclipse Integration Operations feature. Going forward it is planned to support tracing for adapters in the "Monitor Integration Content" feature of the Web application as well.
Finally, in the context of troubleshooting error situations occurring during integration flow execution it is worth to outline two additional rather specific use cases:
(1) In certain situation the message processing runs into an error at a quite early step during inbound processing. At this point in time no message processing log is available. To troubleshoot such error situations you can use the system logs offered for download in the "Monitor System Logs" feature.
(2) In case your integration flow runs into a retry cycle the individual runs of each integration flow execution are displayed in the "Logs" section of the "Monitor Message Processing" feature. Up to 100 runs are listed in the table. You can access the message processing log of each individual run using the hyperlink offered in the "Log Level" column of the table.
I hope you will enjoy using the new capabilities. In case of questions or feedback please feel free to comment on this blog.
References:
Cloud Integration – Setting the Log Level for Message Processing
Cloud Integration - System Log Download
Cloud Integration - Enabling Trace for Message Processing
Avoid Storing Payloads in the Message Processing Log, Especially in Productive Integration Flows
How to Avoid Excessive Storage Load Caused by Using MPL Attachments for Message Logging
Cloud Integration - Apache Camel Exchange Properties recorded as Trace Data
- SAP Managed Tags:
15 Comments
