Cloud Integration – Poll-Enrich with SFTP Adapter for multiple file polling

[fellipe_mendes]

This blog describes a way to configure the Poll Enrich pattern in combination with the SFTP adapter to transfer files from SAP S/4HANA AL11 (DIR_SAPUSERS) to an external SFTP.

Background Information

While working on a SAP S/4HANA implementation project, there was a requirement to transfer files generated on the SAP S/4HANA AL11 (DIR_SAPUSERS) to an external SFTP.
There are some processes that create these files and that need to be transferred to an external SFTP.

Systems and Platforms

SAP S/4HANA FOUNDATION 2020
SAP BTP Integration Suite -> Cloud Integration -> Cloud Foundry 6.30.17
External SFTP – SSH File Transfer Protocol

Business Requirement

Periodically check the SAP S/4HANA folders for new files. When a file is available, it needs to be transferred to the external SFTP server through Cloud Integration and then the file is moved to the archive folder.

High-level process overview

The Challenge with this Integration

With the latest SAP Cloud Integration features, the new Poll Enrich pattern in combination with the SFTP adapter was made available with the main possibilities as mentioned in this blog:

(https://blogs.sap.com/2021/05/12/cloud-integration-use-poll-enrich-with-sftp-adapter/)

- Enrich the message payload with additional data retrieved from a file on the SFTP server
- Poll from a SFTP server triggered by an external trigger, for example triggered via HTTPS call
- Poll from a SFTP server but want to define the configuration in the SFTP adapter dynamically, for example from a partner directory

With this, it is possible to execute a call to an SFTP retrieving a file in the middle of the iFlow, which was previously only possible at the beginning of the iFlow (using the Sender channel).

The challenge begins when Poll-Enrich always polls only one file. If you need to poll multiple files you need to do this Poll-Enrich call in a loop.

However, when you trigger the Poll-Enrich call in a loop, for the first time the file is read (SAP_PollEnrichMessageFound = true); but in the second time, the file is not read  (SAP_PollEnrichMessageFound = false). 

The issue is with the SFTP poll-enrich feature and the post-processing in a loop. 

To avoid this, the recommendation is to implement the loop in one integration flow and from the loop process call another integration flow via ProcessDirect adapter. The second flow then contains the poll-enrich with SFTP, in this way:

Main Flow -> Loop Process Call -> ProcessDirect Adapter -> Second Flow with PollEnrich with SFTP

Solution Implemented 

Two iFLows were created for this solution:

• First iFlow to orchestrate the pooling


• Second iFlow to execute the pooling

First IFlow to orchestrate the pooling

First IFlow to orchestrate the pooling

1. The possibility to trigger the integration scenario on-demand by HTTPS call


2. The possibility to trigger the integration scenario periodically using a Start Timer


3. The main integration process to:

• 
  
  
  
  • Receive the calls (from block 1 or 2) 
  
  
  • Set general properties to control the log level
  
  
  • Set the SFTP address by external parameters
  
  
  • A Groovy script to get the directories to be accessed from a value mapping:
  
  
  • A router to evaluate if has directories or not
  
  
  • A value to start the loop
  
  
  • Call the process that executes the loop
  
  
  • Get the final payload that was combined for each file transferred
  
  
  • Log the final payload (execution log) as an attachment in the message (if configured)
  
  
  
  

      4. A local integration process to transfer all files containing:

• 
  
  
  
  • Get the directory of the current loop stage
  
  
  • A router to check if there is directory or not
  
  
  • Set the SFTP information like Environment, Folder Source and Folder Destination
  
  
  • Define the SFTP environment based on the environment configured
  
  
  • Log the beginning of the process 
  
  
  • Set the loop condition to true
  
  
  • Call the process inside the loop
  
  
  • Log the ending of the process
  
  
  
  

      5. A local integration process to call the next iFlow that pool the file

• 
  
  
  
  • Set the payload that will be transferred to the next iFlow
  
  
  • Request Reply to call the next iFlow 
  
  
  • Get the results of the next iFLow
  
  
  • A router to evaluate if a file was found or not
  
  
  • Log the result of process (If a file was found or not)
  
  
  
  

Second IFlow to execute the pooling

Second IFlow to execute the pooling

1. The main integration process to:

• 
  
  
  
  • Explore the payload received from the previously iFlow with the SFTP informations
  
  
  • A Poll Enrich to fetch the file
  
  
  • A SFTP Sender adapter to fetch the file (any file *)
  
  
  • A router to evaluate if a file was found or not
  
  
  • If found, send to the External SFTP
  
  
  • Fill the body with the results of the execution
  
  
  
  

Execution log

At the end of the process, an execution log will be presented, as following:

Execution log

Benefits/Advantages of the implemented solution

We can find some benefits and advantages, such as:

• 
  
  
  
  • The file transfer will be automatic, not depending on a person for this
  
  
  • An automatic run can be scheduled periodically or also a manual run on demand
  
  
  • Multiple directories
  
  
  • All files in the directory will be transferred 
  
  
  • No need for SAP S/4HANA to communicate directly with the external SFTP
  
  
  • The integration scenario is in the Cloud Integration not influencing the functioning of SAP S/4HANA
  
  
  
  

Possible improvements of the implemented solution

We can find some possible improvements, such as:

• 
  
  
  
  • Transfer files to more than one external SFTP
  
  
  • Add exception routines in case of errors
  
  
  
  

Conclusion

With this solution, it is possible to transfer files from SAP S/4HANA to an external SFTP in a configurable and robust way.

I would be more than happy to hear from our SAP Community about the similar experiences with this requirement.
Please share here in the comments your feedback, thoughts and if you have similar requirements, how you implemented them.

Thank you for reading!!

Fellipe Mendes

[former_member17443]

Thank you for the Great Blog Fellipe Mendes

[fellipe_mendes]

Thanks, avinash_7675

[kanan]

Great blog and is sketched out in a very insightful article. Thanks for sharing.

[David_Davis]

Thanks for the blog. I have one doubt. If S4 is writing the files in AL11 directory then how would CPI pull them? I didn't understand how the solution you designed will read the files from AL11.

[fellipe_mendes]

Thanks, Kanan.

[fellipe_mendes]

Hello abraham2009.

Thank you for your question.

The Basis team provides the connection to the S4 AL11 folder, with this it is possible to configure it in the SAP Cloud Connector and create a virtual host/port for CPI connections.

We have a timer in CPI that will pull the files periodically.

Fellipe

 

[David_Davis]

Hi Fellipe - Thanks for the reply. Could you guide how to connect the AL11 drive to the cloud connector? Any screen grab of the config can help.

[fellipe_mendes]

Hello Abraham.

Unfortunately I don't have many details about this configuration since it is a Basis activity and also the purpose of this blog is to present the necessary configurations in the Cloud Integration to process multiples files using the Poll-Enrich with SFTP Adapter.

I don't know if it helps you but the information I received from the Basis team is that the SFTP is mounted to app servers and added to SAP S/4HANA AL11 (DIR_SAPUSERS).

The configuration in the SAP Cloud Connector is based on the TCP protocol.

Hope this can help you in something.

 

Regards,

Fellipe

[stephankallil]

HI Fellipe. Great blog

Thanks for the clear explanation

 

[former_member650765]

Hello Fellipe,

Very much informative blog. We have implemented it.

Thank you so much.

Also, Can you please provide me the router condition you have applied in HasFile section.

We need to check if the file does not exist,We should trigger another body.

Please suggest.

[fellipe_mendes]

Hello prashanth2009

 

Thank you for your question.

 

The router condition HasFile is based on the standard property value that indicate swhether the file was found or not:

property.SAP_PollEnrichMessageFound

 

HasFIle router

 

Fellipe

[BryanPierce]

Hi. This caught my eye too as the ability to read a file directly from say S/4, or in our case still ECC on premise, has been a struggle with the lack of File Adapters in CPI (both sender and receiver). If I understand the setup as stated above it's really just the SFTP Server sitting on the same system as the connected S/4 server. So, the iFlow is NOT reading from the file system directly, as in using a File Adapter like with PI/PO, but instead is using the SFTP Sender adapter to ready the file from the SFTP Server that sits on the same machine (cloud connected) as S/4. I was excited at first that this might be a work around to the lack of File Adapter... but that's not the case after all. Neat option for the use case stated but doesn't solve the missing File Adapter issue. I know that these adapters are on the road map... well see.