Technology Blogs by Members
Explore a vibrant mix of technical expertise, industry insights, and tech buzz in member blogs covering SAP products, technology, and events. Get in the mix!
cancel
Showing results for 
Search instead for 
Did you mean: 
shashank31191
Explorer
Hi Folks, currently I'm working as Cloud Platform Integration Consultant. I have started exploring scripts in Cloud Platform Integration tool and is trying to achieve things which are not possible using palette provided by SAP. This blog will be talking about one of these scenario, where-in requirement is to poll a File from SFTP server in middle in Integration flow.

Problem scenario:

SFTP adapter offered by SAP Cloud platform Integration only allow polling a file from SFTP server using an SFTP sender adapter.

There could be some requirement, wherein business case is to poll a file in middle of an iflow.

 

Possible solution:

We can use a groovy Script in order to create the connection to SFTP server and then poll the file from there.

This would need :

  1. The SFTP credentials which are deployed into our tenant.

  2. Open Source library, using which we can connect to SFTP server.


 

Accessing SFTP credential deployed in our tenant in a groovy script.

 

To access the credential, SecureStoreService API (Public API to access the deployed user credentials) can be used.

Imports:

com.sap.it.api.securestore.SecureStoreService;

com.sap.it.api.securestore.UserCredential;

 

Method Signature:

public UserCredential getUserCredential(String alias)

 

Return Type:

UserCredential

 

Code Snippet:
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import com.sap.it.api.securestore.SecureStoreService;
import com.sap.it.api.securestore.UserCredential;
import com.sap.it.api.ITApiFactory;

def Message processData(Message message) {

def secureStorageService = ITApiFactory.getApi(SecureStoreService.class, null);
def cred = secureStorageService.getUserCredential(“<Name of SFTP Credential deployed in tenant>”);

if (cred == null){
// error handling
}

//cerd.getUserName() and cred.getPassword method will give the user name and password as String.

return message;
}

 

 

Accessing SFTP server to poll file.

For fetching the file from SFTP server we can use JSch API (Jsch is widely used to connect to SFTP server).

 

Download JSch library from here

 

JSch class serves as a central configuration point, and as a factory for Session objects configured with these settings.

  • Use getSession() to start a new Session.

  • Use one of the addIdentity() methods for public-key authentication.

  • Use setKnownHosts() to enable checking of host keys.

  • See setConfig() for a list of configuration options.


Download the jar from the above mentioned link and upload it in your Integration project workspace.

In case eclipse is the mode of development, then add the jar file after creating package (src.main.resources.lib) or in Web GUI, add the jar file as an Archive in Resources section of the Integration iflow.

 

Below is the sample code snippet which can be used to poll a file from SFTP server. (You can always modify it as per your need)
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;

import com.jcraft.jsch.Channel;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
import com.jcraft.jsch.SftpException;
import com.sap.it.api.securestore.SecureStoreService;
import com.sap.it.api.securestore.UserCredential;
import com.sap.it.api.ITApiFactory;


def Message processData(Message message) {

String userName = “”;
String password = “”;
String hostName = “<Enter the hostname/IP address of SFTP server>”;
String filePath = “<Enter the path of file to be polled>”;
String finalString = "";

def secureStorageService = ITApiFactory.getApi(SecureStoreService.class, null);
def cred = secureStorageService.getUserCredential(“<Name of SFTP Credential deployed in tenant>”);
if (cred == null){
// error handling
}
userName = cerd.getUserName();
password = cred.getPassword();

JSch jsch = new JSch();
Session session = null;
try {
session = jsch.getSession(userName, hostName, 22);
session.setConfig("StrictHostKeyChecking", "no");
session.setPassword(password);
session.connect();

Channel channel = session.openChannel("sftp");
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp) channel;

InputStream stream = sftpChannel.get(filePath);
try {
BufferedReader br = new BufferedReader(new InputStreamReader(stream));
String line;
while ((line = br.readLine()) != null) {
finalString = finalString + line; // You can have your own logic over here
}
} catch (IOException io) {
finalString = io.getMessage();
} catch (Exception e) {
finalString = e.getMessage();
}

sftpChannel.exit();
session.disconnect();
} catch (JSchException e) {
//Exception Handling
} catch (SftpException ee) {
//Exception Handling
}

message.setBody(finalString);

return message;
}

This piece of code will read the mentioned file from SFTP server and will save it in message body of the exchange.

Below is the brief pictorial representation of the integration flow that depicts implementation of the given approach.

 



 

Hope this helps..!! 🙂

 

Using the above code, you can poll a file from SFTP server without using a SFTP adapter.

I'll be writing more on how we can make use of script in Cloud Platform Integration.
32 Comments
Hi Shashank,
Currently there is no official way to poll from sftp in the middle of a flow.
But to add external libraries in the flow may cause trouble when we do an open source update and the integration flows might stop working. So we recommend customers to use standard adapters provided by SAP.

Regards,
Gayathri

 
engswee
Active Contributor
Hi Gayathri

 

Can you elaborate further on why external libraries are discouraged?  And if they are discouraged, why provide such a capability in the first place?

 

Per my understanding, CPI uses the OSGi framework, and each integration flow is deployed as an OSGi bundle. The benefit of using OSGi is that it allows different bundles to work with separate versions of a library. How then would an open source update on the core CPI libraries affect the externally added library?

 

Regards

Eng Swee
mandy_krimmel
Advisor
Advisor
0 Kudos
 

Hello,

I agree that the probability is high that the scenario is not affected, but you can never be sure of the side effects during a open source update on our side.

The scenario you describe may still run after OS update, but it is not officially supported and not tested in SAP regression tests, so SAP does not support this in case you have problems after OS update.

Best regards,

Mandy
engswee
Active Contributor
Hi Mandy

 

Thanks for your response. However "can never be sure" sounds a bit too vague, without any facts to back it up.

 

The capability to add external libraries has been mentioned since the early days of HCI in the following blog - Blog 9: Scripting in Integration Flows. I'm surprised to hear that now SAP is saying it is discouraged, and not supported.

 

SAP has been embracing open source/standards in the recent years, and even so many core parts of CPI are based on open source (Camel, OSGi, Groovy). IMHO, to discourage open source usage in custom CPI development seems counter-intuitive.

 

Regards

Eng Swee
0 Kudos
Hello Shashank,

We have a requirement to read *Control* file from a directory (in ideal situation there will only be one). i tried using the code listed below for this.

sftpChannel.cd(filePath);
Vector<sftpChannel.LsEntry> list = sftpChannel.ls("*Control*");
for(sftpChannel.LsEntry entry : list) {
filename = entry.getFilename();
}
filename=filePath+filename;
InputStream stream = sftpChannel.get(filename);

 

But is ended with error saying unable to resolve class sftpChannel.LsEntry. I checked the jar file imported but it doesn't seem to contain this class. Also i tried alternative downloads for the jsch library, but that did not help.

Please assist.

 

Regards,

Diptee
former_member620584
Discoverer
Is it possible to read file dynamically based on filename,for eg. i have multiple files for order at server and i need to add them back as an attachment to respective order,so how can i get the respective file ?
dirugadr
Participant
0 Kudos
Hello Shashank,

First, thank you for sharing this. It's very useful!

Now, let me ask you something: Is it possible to use this code, in some way, in order to access an SFTP server located into my local network? So far, when I'm using the standard SFTP adapter, I fill the proxy type combo as "on-premise" (and I've configured a connection with Cloud Connector), but now, it seems this code is unable to reach that server.

Regards,

Adrian
Hi Diptee,

Just import this class 'com.jcraft.jsch.ChannelSftp.LsEntry;', it should help.

Regards,
Maciej
0 Kudos
Hello adrian,

 

I have the same requirements (using an on-premise server, where we installed Cloud connector).
So I want to reach the server through a localhost proxy in groovy script.
But everytime I get "fail in SOCKS5 proxy" error.

Did you solve this problem? If so, please contact me or reply.

 

Thanks in advance!

Regards,

Jozsef
dirugadr
Participant
0 Kudos
Hello,

I'm afraid to tell you that I was unable to do it. I had to change the approach I was using in order to complete my task. However, last days I was reading about "custom adapter development", and maybe it could be possible to do it. I'll take a look and I'll let you know my findings.

Regards,

Adrian
Hello,

 

Thanks for your reply.

In the mean time, we managed to make it work.

If you are interested, the solution was:

You had to setup a SOCKS5 Proxy (to localhost), and you have to set the username of the proxy to 1.<subaccount>.<locationId>

 

Code:

final String encodedSubaccount = new String(Base64.encodeBase64(subaccount.getBytes()));

final String encodedLocationId = new String(Base64.encodeBase64(locationId.getBytes()));

com.jcraft.jsch.ProxySOCKS5 proxy = new ProxySOCKS5("localhost", 20004);

proxy.setUserPasswd("1." + encodedSubaccount + "." + encodedLocationId, "");

session.setProxy(proxy);

Regards,

Jozsef
ChrisPaine
Active Contributor
0 Kudos
Hi, it's been a while now since this was raised and still no support for mid flow polling of SFTP locations using standard adaptors (afaik).

I was wondering if there was perhaps an update on whether:

a) SAP had added regression tests to the use of customer libs as part of scripting (a known and hitherto understood as supported approach)? or

b) had road-mapped the enhancement of the  SFTP adaptor to be able to handle these use cases?

Thanks!

Chris
mandy_krimmel
Advisor
Advisor
0 Kudos
Hello,

there is indeed an update. We started working on the feature to support polling files in the middle of the integration flow processing. Still I cannot commit on any timeline but at least we started grooming the feature in more detail and thinking about possible UI options.

Could you maybe share more about the use case you have in mind? Would you like to poll a file and then use this as the new payload or would you rather like to poll a file and then merge it with the existing payload?

Best regars,

Mandy
former_member760
Explorer
0 Kudos
Hi,

 

The use case for me would be that we can have a flow in which some part of the processing of the flow can decide which file to pick up. For example based on the results of an API call decide if we want to pick file A vs file B.

A second use case would be that the run time of a flow would be independant of the arrival of a file.

 

Kind Regards,

 

Marc
mandy_krimmel
Advisor
Advisor
0 Kudos
Hell Marc,

concerning first use case: would you then use the polled file as new payload or would you merge this to the original payload?

concerning second use case: This I dont understand. What is the difference to the sftp sender adapter. There you have a poll interval, and then the file is polled. If nothing is there nothing is polled. With Poll-Enrich you have just a different trigger. But both check on the sftp server for the file and poll it. and in both cases if nothing is there nothing is polled. Maybe you can elaborate a bit more on your use case?

BR,

Mandy
Sourav1
Participant
0 Kudos
perfect. thanks a lot.
we can do a lot with jsch library
0 Kudos
Dears ,

Does still SAP havent any standard way to access files from SFTP server in middle of an iflow?

I have a requirement that i need to integrate with SuccessFactors LMS system to create organization file on the SFTP Server.

I managed to create the iflow and write on the file but i need to know whether the file is already exists or not , as the content would change every time.

 

I mean in the first time the file should be
ID|Name|Date
1 |test|30/8/2020

Then second time the file would be appended and the result will be like

ID|Name|Date
1|test|30/8/2020
2|test2|35/8/2020

 

And this should be done if i managed to know in the middle of the iflow whether the file exists on the server or not so i can decide the content of the file which will be written.
PeterKautz
Newcomer
0 Kudos
Hello all,

does SAP now provide an own solution for polling a file from sFTP server in the middle of an iFlow?

I would like to fetch some files from a sFTP folder but the filename has a timestamp in. So i've to calculate the filename and with the current SAP solution that's not possible. The files are created from Time Management system (Clock-in / clock-out) and to be send to the Employee Central Time component...

regards,
Peter
sinhasouvik
Participant
0 Kudos
As far as I know, There is no standard solution available till now.

 

Regards,

Souvik
lgutiegt
Explorer
0 Kudos
Hi !!

I have tried to take this example as a guide to insert it in a cpi solution, but when trying to connect to the SFPT it throws me the error "Auth Fail".

I have tried to find a solution on the web, but have not been successful. This SFTP Server that I mention is used in other iflow, with its credentials, etc and it works OK. Any idea how to solve my problem.

Thanks
andreschgz95
Participant
0 Kudos
Hello mandy.krimmel

Right now I'm facing an scenario where I need to implement an iFlow that obtains files from SFTP server (replacing the existing payload) on-demand, meaning that obviously I cannot work with the existing adapter since it is required to call the iFlow as you would do with an API in opposition to a job, so I'm using HTTPS to start my iFlow and then putting a groovy script in the middle as shashank31191 points out in this blog. So, at this point everything ok, right?

I have tested the SFTP connection in a Java project in eclipse and everything works fine, but when I'm working inside of CPI's groovy script, I have a problem with the authentication by means of kaypair. I already have implemented the write operation in a different iFlow by using the SFTP adapter and the standard connection to SFTP works fine, but the problem comes when I try to connect using the JSch library.

I'm retrieving the keypair inside my groovy script by using the KeystoreService.getKeyPair method, but what I in fact obtaining by that is the Public and Private key, not the .key file that I originally uploaded to CPI Keystore.

So that's my issue: how can I retrieve the .key file I uploaded into CPI Keystore inside my groovy script to complete the authentication by using addIdentity method of JSch library? That works perfect locally in my Java project (using .key file), but in CPI throws errors like this:

"com.sap.it.rt.adapter.http.api.exception.HttpResponseException: An internal server error occured: No signature of method: com.jcraft.jsch.Session.addIdentity() is applicable for argument types: (java.lang.String, [B, null, [B) values: [salfasftp, [48, -126, 4, -66, 2, 1, 0, 48, 13, 6, 9, 42, -122, ...], ...].
The MPL ID for the failed message is : AGAHG30GgiZ2vEJJi7z_tEr4IzHg
For more details please check tail log."

Hope you have some insight into this problem.

Best regards.
mandy_krimmel
Advisor
Advisor
0 Kudos
As said, the solution which Cloud Integration plans to offer for this requirement is the Poll-Enrich step in combination with the sftp adapter.

This is currently in development and is planned to be shipped in March if all final testing runs through without problems.

How to do this in a script I cannot comment on.

Best regards

Mandy
umehring4cloud
Participant
0 Kudos
Hi Luis,

I had the same issue and was able to solve it by replacing the line

password =cred.getPassword();

with password = new String(cred.getPassword());

 

Hope that helps,

Uwe
former_member185640
Discoverer
0 Kudos
Hi Mandy, does the Poll enrich step is available?
mandy_krimmel
Advisor
Advisor
0 Kudos
Hello

the Poll Enrich step for sftp is currently in integration test and is planned to be released to customers with the update in May.

Best regards

Mandy
pabloreig
Explorer
0 Kudos

Hi Mandy,

Is this feature from the roadmap that will be included in Q2 2021 what you are talking about?

 

Best regards,
Pablo

mandy_krimmel
Advisor
Advisor
0 Kudos
Hello,

yes, this is the one. But in May it will only be available for sftp. ftp will come in a next increment.

Best regards,

Mandy
snehashah
Explorer
0 Kudos
Hi Mandy,

 

Is the Poll-enrich step available for use?
monicabhosale
Active Participant
0 Kudos
Hi Shashank,

 

This is really helpful blog for my current requirement, Will this work for FTPserver as well?

With many thanks

Monica

 

 
former_member31832
Discoverer
0 Kudos

Hi,

After a year does this still without being released?

It's a necessary feature to avoid a bunch of workaround.

 

mandy_krimmel
Advisor
Advisor
0 Kudos
The new Poll Enrich step with sftp Adapter is already available, respective blog is: Cloud Integration – Use Poll-Enrich with SFTP Adapter | SAP Blogs

FTP adapter is planned for one of the next takts.

BR

Mandy
sinhasouvik
Participant
0 Kudos
Hi shashank31191 ,

I have a requirement to send a file to target SFTP folder without using SFTP adapter. Is it possible to achieve this scenario using groovy?

I know that file transfer is possible from one folder to another folder but wanted to know if we want to send the file directly to SFTP using code?

 

Regards,

Souvik
Labels in this area