Replication Flows - SAP Datasphere and Google Big Query

[karishma_kapur]

Background

SAP Datasphere released replication flows starting from SAP Datasphere version 2021.03. This new capability allows one to copy multiple tables from a source to a target in a fast and seamless way. For more information on replication flows, please refer here.

This blog will demonstrate how to replication data from SAP sources to Google BigQuery.

Steps

1. To start, you will need to create a connection in SAP Datasphere to Google Big Query. Please refer to Step 4. “Creating Google BigQuery Connection” in this blog, to make your connection.


2. Make sure you have a Dataset in Google BigQuery that you want to replicate the tables into.


3. Make sure you have a source connection. In this case, we will be using S4 Hana Cloud. You will need to create this connection in the Connections tab in SAP Datasphere.


4. Navigate to SAP Datasphere, and click on Data Builder on the left panel. Find and click the “New Replication Flow” tile.


5. Click on Select Source Connection


6. Choose the source connection you want. We will be choosing SAP S4 Hana Cloud


7. Click select Source Container.


8. Choose CDS Extraction – CDS Views Enabled for Extraction and then click Select.


9. Click “add source objects” and choose the views you want to replicate. You can choose multiple if needed. Once you finalize the objects, click add selection.


10. Now, we select our target connection. We will be choosing Google Big Query as our target. If you experience any errors during this step, please refer to the note at the end of this blog.


11. Next we choose the target container. Recall the dataset you created in Big Query earlier in step 2. This is the container you will choose here.


12. In the middle selector, click to”settings” and set your load type. Initial only means to load all selected data once. Initial and delta means that after the initial load, you want the system to check every 60 minutes for any changes (delta), and copy the changes to the target.


13. Once done, click on the edit projections icon on the top toolbar to set any filters and mapping. For more information on filters and mapping, please refer here, and here.


14. You also have the ability to change the write settings to your target through the settings icon next to the target connection name and container.


15. Finally, rename the replication flow to the name of your choosing in the right details panel. Then, save, deploy, run the replication flow through the top toolbar icons. You can monitor the run in the “data integration monitor” tab on the left panel in SAP Datasphere.


16. When the replication flow is done, you should see the target tables in BigQuery as such. It should be noted that every table will have 3 columns added from the replication flow to allow for delta capturing. These columns are operation_flag, recordstamp, and is_deleted.

Note: You may have to include Premium Outbound Integration block in your tenant to deploy the replication flow.

Conclusion

You have now successfully created a replication flow from SAP S4 HANA Cloud to Google BigQuery. If you have any questions, please leave a comment below. Thank you!

[jack_tee2]

hi karishma_kapur

Great to see the DS has added the hyperscaler connection types. A question to the Replication Flow on the source side, when you run the 4 tables replication. Does it consume (min of) active 4 processors in the origin source system(S4HANA) while this Replication Flow is running?

Thank in advance.

 

[shantanu_a_patil]

Strategically will SAP build a S4HANA Replication connector towards BigQuery directly?  If that happens, will Datasphere be still needed ? and why ?