This blog describes options for configuring asynchronous one-way message processing using the new JMS adapter, which is now available for customers. It describes the setup, configuration, size limits and monitoring of your scenario. The JMS adapter is available for SAP Cloud Integration customers with an Enterprise Edition license since 24-June-2017 release (2.29*).
In many cases integration scenarios have to be decoupled asynchronously between sender and receiver message processing to ensure that a retry is done from the integration system rather than the sender system. This can be achieved, for example, by using JMS queues to temporarily store the messages in the Cloud Integration system if the receiver system cannot be reached, and retry them from there. Follow the steps described below to activate the message broker and to setup the scenario. Use the monitoring tools described to check the processing and view the stored messages in the queue monitor.
To be able to use JMS queues for storing messages you first need to activate the message broker for your cloud integration tenant. If you have an Enterprise Edition license you can do this for your tenant in Cloud Integration Provisioning via Provision Broker. You can see then the active message broker and its details. You can now start using the JMS adapter to configure your scenario.
This is described in detail in blog 'Activating and Managing Enterprise Messaging Capabilities'.
To configure the decoupling of inbound and outbound message processing you need to configure two processes:, one process to receive the inbound message and store it in the JMS queue and a second process to trigger the message from the JMS queue to the receiver backend. The blog describes the configuration using two separate integration flows.
The first integration flow will be configured to receive the message via any inbound adapter. In this sample setup we use the HTTP adapter.
Create the integration flow with the inbound channel required by your scenario, and use the JMS adapter as the outbound adapter. You only have to configure the Queue Name for the JMS queue that will be used to store the messages. It is Important to set an Expiration Period that suits your scenario. The default value is 90 days', which means that after 90 days any messages, that were not delivered within this period will be deleted from the queue.
With the 10-December 2017 release, you will have the option to configure, if exchange properties are forwarded to the JMS queue. Before, the properties were always forwarded along with the message, which caused severe issues in some scenarios. The default value of the option will be that properties are not forwarded, because this could lead to runtime issues if the properties are too large. In detail the size restrictions are described in the Blog 'JMS Resource and Size Limits'.
This option is available in adapter version 1.1.
You need to configure the correct transaction manager in the integration process for transactional end-to-end processing. Our process does not contain any JDBC resources (data store, variables, aggregator), and only one direct JMS Receiver without splitter or sequential multicast; so we don't need a specific transaction handler. Select the integration process and switch to the Processing tab. In Transaction Handling drop down select Not Required.
More details about the different transaction handling options and existing limitations are described in the blog 'How to Configure Transaction Handling in Integration Flow'.
Now you can deploy the integration flow. During deployment, the specified queue will be created automatically in the message broker. You can check that the queue has been created in the Queue Monitor available in the operations view in the Manage Storages section. The monitor is described in more detail in the Monitoring section below.
If a message broker has not yet been provisioned, the deployment will end with an error. In Manage Integration Content the error details for your integration flow tell you that no messaging host is available.
To consume the messages from the JMS queue, you configure a second integration flow with a JMS sender channel and the outbound adapter needed for your scenario. In this sample configuration we use the HTTP adapter.
Create the integration flow with the outbound channel required by your scenario, and use the JMS adapter as the inbound adapter. You only have to configure the Queue Name for the JMS queue that you want to poll the messages from. Use the same queue name used in the receiving integration flow.
Number of Concurrent Processes is set to 1 per default and should only be increased if more parallel processing is needed. Because the JMS resources are limited, you need to keep the number of parallel processes as low as possible. Especially if large messages are processed in the scenario, increasing the parallelism may lead to out of memory problems.
In the JMS sender channel you also have to configure the retry handling. Set the Retry Interval as required for your scenario (, the default value is 1 minute). We also recommend using Exponential Backoff, which means that the retry interval is doubled after each unsuccessful retry. By using this setting you can avoid calling the receiver backend every minute if it is unavailable for a longer time period. The Maximum Retry Interval defines the maximum time between two retries if exponential backoff is used. The recommendation is to either keep the 60 minutes or even increase it if this is acceptable by the scenario.
Note, that the JMS sender does not guarantee the order in which the messages are consumed. If several processes are configured to consume messages and/or multiple worker nodes process messages, the messages are processed in parallel. Furthermore, in case the message is in retry because of an error it is taken out from processing until the retry interval is reached.
With 22-July-2017 release the option to configure a dead letter queue is introduced. This is described in detail in blog How to configure Dead Letter Handling in JMS Adapter.
If you are using JMS Sender, it is not necessary to set the transaction handling to Required for JMS, because the retry handling works independently of the selected transaction handler. You should select the transaction manager as required by the scenario that you have configured in the integration flow.
In the simple example shown, no JDBC or JMS resources are used, that need a transaction handler, so we select Not Required.
There are some limitations on configuring transaction handling configuration and on which flow steps can be used in which combinations. This is described in detail in the blog 'How to configure Transaction Handling in Integration Flow'.
In the JMS sender channel, you can configure the time intervals between retries, but you cannot configure that processing will end after a specific number of retries. If required, you can configure this in an exception subprocess that calls a local process for retry handling using the header SAPJMSRetries set by the JMS sender adapter.
Note that the header SAPJMSRetriesmay not indicate the exact number of retries in clusters with multiple worker nodes. It may happen that a retry is triggered on multiple worker nodes shortly one after the other before the message is moved to the error queue and the counter in the header is increased.
To configure this, add a Local Integration Process to the integration flow. In this process you configure the specific retry handling. In this sample, we check the number of retries executed and after 6 unsuccessful retries, we end the processing and trigger a mail to alert someone to the problem.
In the local integration process configure a Router after the start event, add an Error End Event and configure the router as shown in the picture. The header SAPJMSRetries set by the JMS adapter is used to decide if the message processing continues or if it ends and a mail is triggered. If message processing is to continue, an error is raised by the process so that the message stays in the JMS queue and goes into retry status.
We set the transaction handling to From Calling Process in the local process, and as there are neither JMS nor JDBC transacted resources configured in the local process, we take over the Not Required setting from main process.
Furthermore, you need to add an Exception Subprocess in the main process, and add a Local Process Call, where you select the configured local process, within the exception subprocess. Add a Receiver for the mail to be sent. Connect the Message End Event of the exception subprocess to the receiver using a Mail adapter receiver channel. In this channel, you configure the mail address the mail is to be sent to and the details to be sent.
With this configuration the message will be removed from the JMS queue after 5 unsuccessful retries, so that no further retries are executed. A mail is triggered to notify the appropriate person to take manual action.
If you use the explicit retry configuration, you are free to configure retries as required for your scenario. For example, you can notify someone after 3 retries but not stop the processing until 10 retries have been made, to allow enough time for manual actions. Or, you can send the message after 3 retries via an alternative receiver channel.
You can now deploy the integration flow. This completes the scenario configuration in SAP Cloud Integration and you can start processing messages. Test the configuration by sending a test message via the configured inbound adapter and check that it is received by the receiver backend.
The connected JMS messaging instance, that is used in the asynchronous messaging scenario has limited resources. The Enterprise License sets a limit on the queues, storage and connections that you can use in the messaging instance. The limits are described in more detail in the blog 'JMS Resource and Size Limits'.
If a higher throughput is required for your scenario and you identify JMS as the bottleneck, the following options for performance increase are possible:
To check the processing in detail, you can use the monitoring tools provided by Cloud Integration.
In the operations view, in the section Manage Integration Content, you can find the deployed integration flows and their status. Check that the flows have the status Started.
Poll Status (available with the 16-Feb-2020 update)
In the Status Details area you may find the status and the details about the current consumption:
Message processing can be checked in detail in the section Monitor Message Processing. Select the message you sent and check the processing status. There should be two message processing logs, one for the inbound messaging that saves the message to the JMS queue, and one for processing the message from the queue and sending it to the receiver system.
Those two messages are correlated via a Correlation ID. With the 8th-July-2018 update of Cloud Integration, this Correlation ID is shown in the Properties of a message processing log. You can also search for all messages having the same Correlation ID using the ID search field at the top of the monitor:
In the event of a failure when sending to the receiver system the outbound message will go into status Retry. The message will then be retried as configured in the JMS sender channel. The detailed error information and the retries are available under the link Message Processing Log.
If there are problems during processing and your message does not arrive at the receiver backend, the message status is Retry and the messages can be checked in the Queue Monitor in the section Manage Storages in the Operations view. This monitor shows all queues existing in your tenant, with their stored messages. Select your queue to check the messages contained in the queue.
You can see the number of retries that have already been executed and the time of the next retry. You can also trigger an immediate retry of the message or delete specific messages if you no longer want them to be processed.
If a message broker has not yet been provisioned, the Queue Monitor tile is not shown.
If you do no longer need a specific queue, (for example if you un-deploy the integration flow), you have to delete the queue manually in the Queue Monitor. Unused queues, that do not contain messages will be deleted automatically after the May-13-2018 update. Queues containing messages cannot be deleted automatically because this could cause inconsistencies, as the messages in the queue may still have to be processed.
To find unused queues in the monitor you may use the checks described in the blog 'Checks in JMS Message Queue Monitor'.
Asynchronous Messaging with Retry Using the JMS Adapter
In many cases integration scenarios have to be decoupled asynchronously between sender and receiver message processing to ensure that a retry is done from the integration system rather than the sender system. This can be achieved, for example, by using JMS queues to temporarily store the messages in the Cloud Integration system if the receiver system cannot be reached, and retry them from there. Follow the steps described below to activate the message broker and to setup the scenario. Use the monitoring tools described to check the processing and view the stored messages in the queue monitor.
Prerequisite: Broker Provisioning
To be able to use JMS queues for storing messages you first need to activate the message broker for your cloud integration tenant. If you have an Enterprise Edition license you can do this for your tenant in Cloud Integration Provisioning via Provision Broker. You can see then the active message broker and its details. You can now start using the JMS adapter to configure your scenario.
This is described in detail in blog 'Activating and Managing Enterprise Messaging Capabilities'.
Setup Scenario With Asynchronous Decoupling
To configure the decoupling of inbound and outbound message processing you need to configure two processes:, one process to receive the inbound message and store it in the JMS queue and a second process to trigger the message from the JMS queue to the receiver backend. The blog describes the configuration using two separate integration flows.
Configure the Integration Flow Receiving the Message
The first integration flow will be configured to receive the message via any inbound adapter. In this sample setup we use the HTTP adapter.
Configure the JMS Receiver Channel
Create the integration flow with the inbound channel required by your scenario, and use the JMS adapter as the outbound adapter. You only have to configure the Queue Name for the JMS queue that will be used to store the messages. It is Important to set an Expiration Period that suits your scenario. The default value is 90 days', which means that after 90 days any messages, that were not delivered within this period will be deleted from the queue.
With the 10-December 2017 release, you will have the option to configure, if exchange properties are forwarded to the JMS queue. Before, the properties were always forwarded along with the message, which caused severe issues in some scenarios. The default value of the option will be that properties are not forwarded, because this could lead to runtime issues if the properties are too large. In detail the size restrictions are described in the Blog 'JMS Resource and Size Limits'.
This option is available in adapter version 1.1.
Configure Transaction Handling
You need to configure the correct transaction manager in the integration process for transactional end-to-end processing. Our process does not contain any JDBC resources (data store, variables, aggregator), and only one direct JMS Receiver without splitter or sequential multicast; so we don't need a specific transaction handler. Select the integration process and switch to the Processing tab. In Transaction Handling drop down select Not Required.
More details about the different transaction handling options and existing limitations are described in the blog 'How to Configure Transaction Handling in Integration Flow'.
Deploy the Integration Flow
Now you can deploy the integration flow. During deployment, the specified queue will be created automatically in the message broker. You can check that the queue has been created in the Queue Monitor available in the operations view in the Manage Storages section. The monitor is described in more detail in the Monitoring section below.
If a message broker has not yet been provisioned, the deployment will end with an error. In Manage Integration Content the error details for your integration flow tell you that no messaging host is available.
Configure the Integration Flow Doing the Retry
To consume the messages from the JMS queue, you configure a second integration flow with a JMS sender channel and the outbound adapter needed for your scenario. In this sample configuration we use the HTTP adapter.
Configure the JMS Sender Channel
Create the integration flow with the outbound channel required by your scenario, and use the JMS adapter as the inbound adapter. You only have to configure the Queue Name for the JMS queue that you want to poll the messages from. Use the same queue name used in the receiving integration flow.
Number of Concurrent Processes is set to 1 per default and should only be increased if more parallel processing is needed. Because the JMS resources are limited, you need to keep the number of parallel processes as low as possible. Especially if large messages are processed in the scenario, increasing the parallelism may lead to out of memory problems.
In the JMS sender channel you also have to configure the retry handling. Set the Retry Interval as required for your scenario (, the default value is 1 minute). We also recommend using Exponential Backoff, which means that the retry interval is doubled after each unsuccessful retry. By using this setting you can avoid calling the receiver backend every minute if it is unavailable for a longer time period. The Maximum Retry Interval defines the maximum time between two retries if exponential backoff is used. The recommendation is to either keep the 60 minutes or even increase it if this is acceptable by the scenario.
Note, that the JMS sender does not guarantee the order in which the messages are consumed. If several processes are configured to consume messages and/or multiple worker nodes process messages, the messages are processed in parallel. Furthermore, in case the message is in retry because of an error it is taken out from processing until the retry interval is reached.
With 22-July-2017 release the option to configure a dead letter queue is introduced. This is described in detail in blog How to configure Dead Letter Handling in JMS Adapter.
Configure Transaction Handling
If you are using JMS Sender, it is not necessary to set the transaction handling to Required for JMS, because the retry handling works independently of the selected transaction handler. You should select the transaction manager as required by the scenario that you have configured in the integration flow.
In the simple example shown, no JDBC or JMS resources are used, that need a transaction handler, so we select Not Required.
There are some limitations on configuring transaction handling configuration and on which flow steps can be used in which combinations. This is described in detail in the blog 'How to configure Transaction Handling in Integration Flow'.
Optional: Explicit Retry Configuration
In the JMS sender channel, you can configure the time intervals between retries, but you cannot configure that processing will end after a specific number of retries. If required, you can configure this in an exception subprocess that calls a local process for retry handling using the header SAPJMSRetries set by the JMS sender adapter.
Note that the header SAPJMSRetriesmay not indicate the exact number of retries in clusters with multiple worker nodes. It may happen that a retry is triggered on multiple worker nodes shortly one after the other before the message is moved to the error queue and the counter in the header is increased.
To configure this, add a Local Integration Process to the integration flow. In this process you configure the specific retry handling. In this sample, we check the number of retries executed and after 6 unsuccessful retries, we end the processing and trigger a mail to alert someone to the problem.
In the local integration process configure a Router after the start event, add an Error End Event and configure the router as shown in the picture. The header SAPJMSRetries set by the JMS adapter is used to decide if the message processing continues or if it ends and a mail is triggered. If message processing is to continue, an error is raised by the process so that the message stays in the JMS queue and goes into retry status.
We set the transaction handling to From Calling Process in the local process, and as there are neither JMS nor JDBC transacted resources configured in the local process, we take over the Not Required setting from main process.
Furthermore, you need to add an Exception Subprocess in the main process, and add a Local Process Call, where you select the configured local process, within the exception subprocess. Add a Receiver for the mail to be sent. Connect the Message End Event of the exception subprocess to the receiver using a Mail adapter receiver channel. In this channel, you configure the mail address the mail is to be sent to and the details to be sent.
With this configuration the message will be removed from the JMS queue after 5 unsuccessful retries, so that no further retries are executed. A mail is triggered to notify the appropriate person to take manual action.
If you use the explicit retry configuration, you are free to configure retries as required for your scenario. For example, you can notify someone after 3 retries but not stop the processing until 10 retries have been made, to allow enough time for manual actions. Or, you can send the message after 3 retries via an alternative receiver channel.
Deploy the Integration Flow
You can now deploy the integration flow. This completes the scenario configuration in SAP Cloud Integration and you can start processing messages. Test the configuration by sending a test message via the configured inbound adapter and check that it is received by the receiver backend.
JMS Resource and Size Limits
The connected JMS messaging instance, that is used in the asynchronous messaging scenario has limited resources. The Enterprise License sets a limit on the queues, storage and connections that you can use in the messaging instance. The limits are described in more detail in the blog 'JMS Resource and Size Limits'.
Performance Optimization Options
If a higher throughput is required for your scenario and you identify JMS as the bottleneck, the following options for performance increase are possible:
- Do not use intermediate JMS queues in one end-to-end scenario. The recommendation is to use one queue after the message is received from the sender system and one directly before the message is sent to the receiver system. Depending on the scenario, one JMS queue could be sufficient as well.
- Increase the number of parallel processes in the JMS sender channel to make sure more messages are consumed in parallel. But keep an eye on the JMS resources, see blog 'JMS Resource and Size Limits'.
- Switch off Encryption in the JMS receiver channel. For scenarios without sensitive data this option could be used to increase performance.
- Do not use the Dead-Letter Queue. The option to switch off the dead-letter queue you should only use if you can make sure that no out-of-memory errors can occur, for example because of large messags. If you have only small messages and don't see a risk for out-of-memory situations in your scenario switching off the Dead-Letter Queue could increase the performance. More details can be found in the blog How to configure Dead Letter Handling in JMS Adapter.
- Use the correct transaction handling. It is for example important to use a JMS transaction handler if JMS sender and JMS receiver channels are used in the same integration process. More details can be found in the blog 'How to configure Transaction Handling in Integration Flow'.
Monitoring
To check the processing in detail, you can use the monitoring tools provided by Cloud Integration.
Integration Flow Monitoring
In the operations view, in the section Manage Integration Content, you can find the deployed integration flows and their status. Check that the flows have the status Started.
Poll Status (available with the 16-Feb-2020 update)
In the Status Details area you may find the status and the details about the current consumption:
Message Monitoring
Message processing can be checked in detail in the section Monitor Message Processing. Select the message you sent and check the processing status. There should be two message processing logs, one for the inbound messaging that saves the message to the JMS queue, and one for processing the message from the queue and sending it to the receiver system.
Those two messages are correlated via a Correlation ID. With the 8th-July-2018 update of Cloud Integration, this Correlation ID is shown in the Properties of a message processing log. You can also search for all messages having the same Correlation ID using the ID search field at the top of the monitor:
In the event of a failure when sending to the receiver system the outbound message will go into status Retry. The message will then be retried as configured in the JMS sender channel. The detailed error information and the retries are available under the link Message Processing Log.
Queue Monitor
If there are problems during processing and your message does not arrive at the receiver backend, the message status is Retry and the messages can be checked in the Queue Monitor in the section Manage Storages in the Operations view. This monitor shows all queues existing in your tenant, with their stored messages. Select your queue to check the messages contained in the queue.
You can see the number of retries that have already been executed and the time of the next retry. You can also trigger an immediate retry of the message or delete specific messages if you no longer want them to be processed.
If a message broker has not yet been provisioned, the Queue Monitor tile is not shown.
Queue Deletion
If you do no longer need a specific queue, (for example if you un-deploy the integration flow), you have to delete the queue manually in the Queue Monitor. Unused queues, that do not contain messages will be deleted automatically after the May-13-2018 update. Queues containing messages cannot be deleted automatically because this could cause inconsistencies, as the messages in the queue may still have to be processed.
To find unused queues in the monitor you may use the checks described in the blog 'Checks in JMS Message Queue Monitor'.
- SAP Managed Tags:
70 Comments
