Does SAP Data Intelligence Store data that passes through the graphs?

[selvakumar_mohan]

We have a requirement to send data from SAP S4 on prem to GCP big query. We are evaluating SAP Data Intelligence as tool for the same. We have a question around the same, which is, does the data(from any data source to any destination) that passes through these graph, do they get stored in any interim tables(Within DI) between source and destination

[jimgiffin · Accepted solution]

There is a Gen2 Cloud Table Producer that will push data into GBQ - see the documentation here: https://help.sap.com/docs/SAP_DATA_INTELLIGENCE/97fce0b6d93e490fadec7e7021e9016e/f6679497feba49189a9...

It does use a temporary file storage for staging in GCS but it is fully managed by the operator.

[selvakumar_mohan · Accepted solution]

This was dearly helpful for our situation, but do you have any documentation which can help us set up the Staging area documentation?

[jimgiffin · Accepted solution]

Selvakumar,

For GBQ, you just need to create a GCS storage connection, select the connection in the "Staging Connection ID". Once you do that, you can navigate to the path you want to use within GCS to stage the files. In my testing I used "/staging". The operator will drop files there temporarily and remove them once they are loaded. Note that the table in GBQ must be created already for the mapping in the operator.

[selvakumar_mohan · Accepted solution]

Hi James, this worked out in addition to "Read data from SAP" operator. I used "Read data from SAP" & "Cloud Table producer". Do you know what to use when data sources are other than "CDS". Is there a Gen 2 operator which can read from ODP extractors?

[jimgiffin · Accepted solution]

At this time there is no ODP in gen2 operators. See details in this overview note: https://launchpad.support.sap.com/#/notes/2890171

However, it looks to be on the roadmap for Q1 2023:

[S0023888701]

Hello Mohan,

SAP DI will not store any data while in transit through DI pipelines unless you make it to persist.

Looking at your requirement, I am not sure SAP DI have direct connectivity to GCP Bigquery but SAP Data service has direct connector to GCP Bigquery. You can refer Export data from SAP systems to Google BigQuery through SAP Data Services | Google Cloud.

SAP DI is your only option and if Bigquery connector option is not available then you can explore below approach.

1. S4-->SAP DI--> GCP Pub/Sub

2. GCP Pub/Sub-->Cloud DataFlow--> Bigquery

In case SAP DI has Bigquery Connector then you are sorted.

Hope this helps!

[selvakumar_mohan]

Yes, they do have a GCP Big query table connector now. We are still exploring the capability but not 100% sure yet. But I will update here when I get that information.